How To Use a Robots.txt File

How To Use a Robots.txt File

Exactly what is a robots.txt and what does it do?

The official Google definition of robots.txt is : “A robots.txt file tells search engine crawlers which pages or files the crawler can or can’t request from your site. This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google.

How To Use a Robots.txt File

by BeBizzy Consulting | BeBizzy Break Podcast

How To Use a Robots.txt file on your webiste

What is a Robots.txt file?

    • Basically, it’s just a file on the root of your website that tells search engines (Google, Bing, Yahoo) what pages and files to index, and what to ignore.
    • If properly coded, a robots.txt file will prevent images, videos, audio files, and even script and style files from being indexed. 
    • HTML and other website page filetypes can be excluded as traffic management. If you actually want to block a page from search results either password protect them, require authentication, or use a noindex directive on the page itself.
    • If you use a hosting service or pagebuilder you might not have a way to provide a robots.txt file. Many will have a way to discourage search engines from indexing the site.

What Are some Examples of Robots.txt Use?

    • The syntax for a robots.txt file is 
      • User-agent : Google, Bing, * (wildcard)
      • Instruction : Disallow, Crawl-delay, Sitemap
      • Rules are case sensitive, so be careful
      • The default setting is that any search engine can index the entire website, so robots.txt provides some directions to enhance or change that.
    • Any number of “Groups” can be created. 
      • Groups is an easy way to separate multiple engine instructions 
        • Group 1 – Google not allowed so index a certain directory
        • Group 2 – All other engines allows to search entire site
    • For example, this syntax would block all search engines from all content (notice the addition after Disallow)
      • User-agent: * Disallow: /
    • And this syntax would ALLOW all search engines to index all content 
      • User-agent: * Disallow:
    • Block a specific search engine from a specific page
      • User-agent: Bingbot Disallow: /example-subfolder/blocked-page.html

What is Crawl Budget?

    • Many tools and resources will mention the “crawl budget” of a website. Basically it’s a number known only to the search engine on how many pages, images, and other files the engine will index, or how long an engine will stay on a site. 
    • If you think pages aren’t being fully indexed, it may be a good idea to identify the page you absolutely need to have indexed and set them to allow (Disallow:). That way the search engines will look at them first.

Why would I use crawl-delay?

    • A directive command that can be used is crawl-delay, then a second command. 
    • Crawl delays will slow down a search engine like Bing, which tends to be a little quick to start. This can increase accuracy while decreasing the load on the site and bandwidth. 
    • Heads up, Google does not use the crawl-delay directive.
    • (crawl-delay: 10)


Common Robots.txt Rules

Disallow crawling of the entire website. Keep in mind that in some situations URLs from the website may still be indexed, even if they haven’t been crawled.This does not match the various AdsBot crawlers, which must be named explicitly.

User-agent: *

Disallow: /

Disallow crawling of a directory and its contents by following the directory name with a forward slash. Remember that you shouldn’t use robots.txt to block access to private content: use proper authentication instead. URLs disallowed by the robots.txt file might still be indexed without being crawled, and the robots.txt file can be viewed by anyone, potentially disclosing the location of your private content.

User-agent: *

Disallow: /calendar/

Disallow: /junk/

Allow access to a single crawler

User-agent: Googlebot-news

Allow: /

User-agent: *

Disallow: /

Allow access to all but a single crawler

User-agent: Unnecessarybot

Disallow: /

User-agent: *

Allow: /

Disallow crawling of a single web page by listing the page after the slash:

User-agent: *

Disallow: /private_file.html

Block a specific image from Google Images:

User-agent: Googlebot-Image

Disallow: /images/dogs.jpg

Block all images on your site from Google Images:

User-agent: Googlebot-Image

Disallow: /

Disallow crawling of files of a specific file type (for example, .gif):

User-agent: Googlebot

Disallow: /*.gif$

Disallow crawling of an entire site, but show AdSense ads on those pages, and disallow all web crawlers other than Mediapartners-Google. This implementation hides your pages from search results, but the Mediapartners-Google web crawler can still analyze them to decide what ads to show visitors to your site.

User-agent: *

Disallow: /

User-agent: Mediapartners-Google

Allow: /

To match URLs that end with a specific string, use $. For instance, the sample code blocks any URLs that end with .xls:

User-agent: Googlebot

Disallow: /*.xls$



Viewing of mobile websites has increased from over 30% in 2015 to now over 50%, and there’s no sign of slowing down. Even if your customers are thought to be mainly on desktop and laptop computers, mobile indexing will force you to get your website designed for mobile use starting in March 2021. 

If you need help getting this process done, especially in a WordPress environment, please contact BeBizzy Consulting at and let’s get your site ready for mobile use.

Thanks for listening to this episode of the WP Wednesday Podcast

Do you have questions, experiences related to today’s topic? Head over to @Bebizzy on Twitter and send them there. 

Don’t forget to check out SEM Rush for all your SEO needs. Visit

And remember to subscribe to the WP Wednesday Podcast for more great tips on managing your WordPress website.

Then, click in your podcast player to subscribe and leave us a review. Then you can sit back, relax, and leave the technical stuff to us.

WordPress News

  • WordPress 5.7 Released March 9, 2021
    • Reusable Blocks
    • Easier font-size adjustments
    • Drag and drop from the inserter right into your page or post
    • Switch from HTTP to HTTPS in one click. No database edits
    • Lazy loading of iFrames

How To Use a Robots.txt File

by BeBizzy Consulting | BeBizzy Break Podcast

Does Google Value Hidden Content Less?

Does Google Value Hidden Content Less?

Hidden Website Content

A fairly common way to display large chunks of content on a website is to use tabs, accordions and toggles. All of these “hide” content until they are clicked or otherwise activated, revealing the content underneath.

So how does Google index this content, or even DOES it index this content?

Does Google Value Hidden Content Less?

by BeBizzy Consulting | BeBizzy Break Podcast

Why Would You Hide Content on a Website?

  • Too much content
  • Want a call to action from the visitor/reader
  • Not something everyone who visits the page needs to see

Does Google Read It If It’s Hidden?

One side says that if you have hidden content, Google doesn’t care and it indexes it just like “normal” content. The other side says if it’s not visible, Google will NOT index it, thereby not including it in searchable content.

I am in the camp of it WILL be indexed and searchable. It’s HTML, code, content on the page, so Google will index it. Search engines read popups, meta data, alt tags, etc, so why not content that is technically on the page?

Matt Cutts

Matt Cutts is a former Google employee as part of the search quality team on search engine optimization issues and also the former head of the web spam team at Google. Her’e’s what he has to say about hidden content.

The main point of Matt’s video is make sure the content being hidden is meaningful or relevant to the page’s main content. Otherwise it could be taken as keyword stuffing.

John Mueller

John Mueller is the host of Google Webmaster Central and a Senior Webmaster Trends Analyst at Google. He backs up what Matt said earlier.

These two experts back up my contention that hidden content on the page will be indexed as long as it’s good content… just like everything else on a web page.

Tabs, toggles and other forms of displaying layered content is common now, so everyone knows how to use them. Use these tools wisely and you won’t have any trouble getting found on the search engines.

Does Google Value Hidden Content Less?

by BeBizzy Consulting | BeBizzy Break Podcast

Fonts In WordPress

Fonts In WordPress

Using Fonts in WordPress

Most websites have images. Some have videos. Others have database connections, feeds from other sites, shopping carts, and others item.

But it’s fair to say that EVERY websites has fonts. From the classic Times New Roman, to newly launched fonts like Shapiro, fonts are text stylings used to display written text on sites.

There are many ways to edit font types but today we’re going to talk about how you can add or change the fonts on your site.

Fonts in WordPress

by BeBizzy Consulting | BeBizzy Break Podcast

Fonts in WordPress are a great way to stylize your website. Changing the font can make a site look very corporate or newsy, or fun and light. Therefore fonts can make a site hard to read or amateurish, so choose wisely.

Most sites use primarily two fonts, one for headings, and one for body text. There are often other fonts used for beneath images, advertisements, and other special cases, but two is the base number needed.

Need some suggestions on font pairings for your site? Check out Font Pair to see fonts paired together and how they will look in WordPress.

Finding Fonts

There are hundreds of fonts in WordPress available online for free or at a very low cost.

Make Your Own Font!

A site called Font Squirrel will take a font you own, upload it to the application, and edit to your needs. Add tails to letters, increase the boldness,

Use the Fonts in WordPress

Once you’ve decided on which fonts will work on your site, it’s time to implement them. There are many ways including CSS, themes and plugins.

  • Themes Some themes already have a customizer in their options to set default heading and paragraph fonts. This makes selecting and changing them extremely easy, as you only have to edit in one place and it should universally change the pages.Of course if any changes were made inside a page they may still hold the values. I use the DIVI theme and changing fonts sitewide is very simple, and using a visual builder it’s quick to see a change in both font style and size will affect the site.
  • Plugins Like setting a font in WordPress in the theme customizer, plugins offer an easy way to change fonts sitewide. Most will interface directly with Google Fonts and other depositories to bring in the fonts and other styles.
  • CSS The most powerful, the fasted loading time, and maybe the most difficult way to adjust your fonts is through CSS. CSS is some instructions on how your WordPress and other site display text, images, page width, navigation and many other options on the site.

To set the fonts and other properties in CSS you will need the following
1- Downloaded font files, or in some cases have code from a depository (like Google Fonts) which can provide an “@import” instruction to pull the files needed in your CSS page/site.
2 – Access to edit the WordPress files to bring in the fonts through CSS. Sometimes this will be a custom CSS file. Other times it will be a specific file in the WP theme. Find out where to change on your page.
3- Sometimes changes to fonts can be made in the header.php file. Place the font import in the <HEAD></HEAD> portion of the HTML.
4- If you host the font files locally, the “@font-face” CSS command can be used.

See more ways and even get specific CSS and HTML code to add fonts in WordPress by visiting this article on


  • WP Beginner – How to Change MX Records For Your WordPress Site

MX Records are where your email is directed. Often times it is the same server as the web host, but more advanced sites use an ARecord host (website), and MX Record host (mail) at different locations. This article gives some great instructions on why, and how, you would change the server records.

  • WP Lift – How to Optimize Cumulative Layout Shift on WordPress

This one might be slightly advanced. If you’ve been using page scoring sites like GTMetrix or Page Insights lately, you’ve probably noticed one of the big items listed as a negative is Cumulative Layout Shift.

It’s when the code is loading and the site layout flashes as images, videos and other content fall into place on the site. It can be distracting, but there are ways around it. If the site loads quickly it may not be an issue, but if you’re concerned about this the page gives some great ideas on how to minimize the shift.

UPDATE : You can use legacy reporting from GTMetrix ( )F

  • Search Engine Journal – WordPress 5.6 Guidance on PHP 8 Compatibility

The most recent version of PHP (7.4) doesn’t reach it’s projected EOL (end of life) until Dec 2022, so if you’re running near the most recent version everything should be fine… for now. But if you like to live on the technical edge, PHP 8 is starting to roll out and WordPress 5.6 will be compatible in beta form, meaning running it on a production website is risky at best.

There are bound to be issues with the core, and almost certainly will be with themes, plugins and even custom code. So implement at your own risk.


Fonts in WordPress

by BeBizzy Consulting | BeBizzy Break Podcast

Great Website Hosting : InMotion Hosting

Great Website Hosting : InMotion Hosting

Choosing Your Website Host is Critical

There are so many pieces to a great WordPress website. The WP version, PHP, theme, plugins, images, content, videos… all contribute to the website’s performance.

But even if everything is optimized as much as possible, if your server isn’t performing, the site will still not serve as quickly as possible.

I’ve tried many server hosts in the last decade, and my favorite so far is undeniably  InMotion Hosting.

InMotion Hosting

by BeBizzy Consulting | BeBizzy Break Podcast

A Little About InMotion Hosting

Joining me on the BeBizzy Break Podcast toay was Robert Hood, Assistant Product Line Manager for both WordPress and Dedicated Hosting.   

InMotion Hosting is a hosting company based in Virginia Beach, VA. They do have satellites in Los Angelas and Denver.

Support Is a Focus for InMotion

  • Agents are trained by agents who have been on the support floor for months, even years.
  • Agents are continuously trained to support new issues or to better handle customer requests.

Some technical info

  • All hard drives are Samsung SSDs
  • Intel-based CPUs
  • CentOS operating systems. No Windows support at this point. 
  • Redundant, secure server locations in Los Angelas and Virginia.
  • Caching – NGNIX
    • NGNIX is a proxy-cache, so the site is served from memory instead of the hard drive. 
    • NGNIX can be turned on or off.

What Are People Looking For In A Server Host

  • Price
  • Technology
  • Support – they are there 24×7, 365 days.
  • Uptime, as for most hosts is 99.9% uptime goal.
    • Dedicated staff of support people watching the server environment at all times controlling 

Hosting Types

Hosting is similar to where you live. 

  • Shared Hosting – Apartment. Communal system sharing resources like water, power, trash. You are dependent on everyone else on that server to be responsible for bandwidth and malware.
  • VPS Hosting – Townhome. Technically connected, but your own water, power, privacy, etc. Your site can still be dependent on others being responsible, but you are insulated a bit.
  • Dedicated Hosting – Private home. You have control over basically everything. You have your server resources, your bandwidth, etc.

Support Types

  • Basic Support
    • Available 24×7 – 265
    • Handles almost 95% of the issues you encouter
  • Advanced Product Support (Robert used to be the manager of this support team)
    • Junior system admins
    • Higher degree of training
    • Assigned to VPS-3000 customer or dedicated box
    • They have resources available to help, but generally an agent owns an issue from start to finish; no hand-offs.
  • Paid Support Teams
    • WTR Team – Website Transfer Team
      • Get some time with that when you spin up a new server
      • Available at later times for more transfers
  • Managed Hosting Team
    • For-hire system admins
    • Bye per-hour or subscription programs 
    • Help with setup, custom configuration, advanced issues.

My experience with InMotion Hosting has been incredible. They are competitively priced, have updated technology, and superior support. If you’re looking for a better host for your WordPress site, InMotion Hosting is my recommendation. 

Where are you hosting your website? Leave it @BeBizzy on Twitter!

InMotion Hosting

by BeBizzy Consulting | BeBizzy Break Podcast

Speeding Up Your WordPress Site With WP-Rocket

Speeding Up Your WordPress Site With WP-Rocket

What is WP Rocket?

WP Rocket is a caching and optimization plugin that takes the code, content, images, videos and other pieces of your website and finds a faster way to present them to your website visitors. That can be storing them on their computers or on a CDN (content data network). It can be combining or moving CSS and javascript files to minimize the time spend processing the code. It can be reducing the number of calls made off the server to Google or other sites.

WP Rocket does all of this, with about ten minutes of setup and a few bucks. Speaking of a few bucks… if you want to purchase WP Rocket, go to and get started today!

WP Rocket

by BeBizzy Consulting | BeBizzy Break Podcast

How Can WP Rocket Speed Up My Website?

There’s only so much you can do technically to increase your website speed. Load times are based on server response times, bandwidth, hard drive speeds… and then it’s down to the software. 

CDNs and optimized code definitely help but eventually you’re down to leaning on more involved 


  • Basic setup was done in about 10 minutes.  Advanced setup, configuration and testing can take additional time, but all options are pretty much toggled on/off.
  • Page caching
    Really decreases page load time by serving up indexed and cached items.
  • Preloading
    Because the crawler simulates a visit to preload the cache, the indexing of your website by search engines is instantly improved.
  • Sitemap loading
  • GZIP Compression
    WP Rocket compresses HTML, CSS, javascript and other code to optimize it by decreasing the size.
  • Browser Caching
    Modern browsers can look for the same photo, video, HTML code and other pieces of a website faster locally that it can sending requests to a website. WP Rocket helps optimize that.
  • Database optimization
    WP Rocket helps clean up databases and make them run more efficiently.
  • Google fonts
    Decreases the requests for unused fonts by only grabbing what is needed to properly display on the site.
  • Remove query strings from Static Resources
    Improves load time and testing scores by removing query strings from CSS and JS files.
  • Lazyload images and videos
    Images and other content doesn’t load until it becomes evident the visitor wants or needs them.
  • Defer Javascript
    Load JS files at the bottom of the page unless necessary elsewhere. This speeds up the page by not loading code before visible items.
  • CDN
    I have mine connected to my Stackpath CDNs, but Cloudflare and other URLS are easily inserted in the settings.
  • Works with eCommerce


All pricing for WP Rocket is on an annual basis to keep updates and support access. You could allow the plugin to lapse, but as WordPress, PHP versions, your themes, your other plugins and more advance the plugin may not work as intended and also become a security issue.

  • $49 for one site
  • $99 for three sites (annual)
  • $249 for unlimited websites

Some other features

  • Works for multi-lingual sites
  • Works in multisite environments
  • You can turn on and off features to connected users with a simple toggle button
  • All features of the plugin can work on mobile without additional programming or configuration.

Support is email support. Only sent one ticket but got a response in about two hours, which is decent.

If you’re looking to speed up your WordPress site, I can’t recommend WP Rocket enough! Head over to and get signed up.

Do you use WP Rocket or another caching pluging? Leave me comments @BeBizzy on Twitter!

WP Rocket

by BeBizzy Consulting | BeBizzy Break Podcast