How to Stop Google from Indexing Your WordPress Site

How to Stop Google from Indexing Your WordPress Site

The whole idea of this post might seem crazy. After all, don’t you want search engines to find your site? That’s why you optimize it for search and why you create an XML sitemap to improve your chances of ranking in search.

That said, there are a number of reasons why you might want to stop Google from indexing your WordPress site:

  • Brand new WordPress installations: It’s not likely that a completely barren and un-optimized site will be crawled and indexed by search engines the second after you’ve installed WordPress. However, you still won’t want to run the risk of someone stumbling upon your incomplete live domain while it’s a work in progress.
  • Development sites: Dev sites, testing sites, staging sites—you set these up so that team members or clients could test out the new (or rebranded) site in real time. You obviously don’t want anyone else stumbling upon a site that’s obviously not ready to be shared with the world.
  • Private sites: If you’ve created something like a company intranet or a private blog (or simply have certain pages you don’t want seen by others), you might also have good reason to want to stop Google from indexing your WordPress site.
  • Non-essential pages: It probably isn’t a huge deal if someone should stumble across your “Thank You” page or some other non-essential that isn’t worth indexing, but know that there are ways to block these as well.

So, once you’ve decided that you want to stop search engines from indexing your site, how do you let them know to stay away? Here are six options to consider, based on what your end goal is.

Preventing Google from Indexing Your WordPress Site

Stopping Google from indexing your WordPress site is a relatively easy process. Before selecting one of the options below, however, make sure you know which pages or posts (if not the entire site) you want to block. You’ll also want to figure out if those pages have already been indexed as that will require a completely different action.

Here are your options:

Option 1: Use WordPress Settings

This is the easiest of the options to implement and likely one you’ve seen in your travels through WordPress.

In order to block an entire site from search, go into WordPress and find the Settings menu. Then select Reading.

Scan down to the option for Site Visibility and you’ll see this:

WordPress Settings – Reading

This, in essence, adds a “noindex” tag to the head tag of your site:

<META NAME="robots" CONTENT="noindex">

However, as WordPress notes, search engines may still choose to go in and index your site anyway. Which is why you should consider doing option #2 as well.

Option 2: Password-Protect the Root

This option will take you out of WordPress and into your site’s control panel. Here is how you’ll do it:

Log into your web hosting account and open the control panel. You should see something called “Directory Password” or “Password Protect Directories”.

You’ll arrive at a page that shows you all the directories on your site. Click on the root directory (“/”).

This final step will then give you the ability to lock up the root of your website, essentially blocking Google (as well as everyone else) from accessing your site.

You can also grant access to specific people—like members of your team who need to get into the site while it’s in progress—through this page.

Option 3: Use the robots.txt File

Now, if you’d rather not block your entire site from search, but instead want to restrict it to certain parts of your site, you can go directly into your robots.txt file in order to apply the block.

Here is the command to enter:

User-agent: *
 Disallow: /[slug]/

Don’t forget to update the slug with the target URL path you want to block.

Option 4: Use a Plugin to Block Your Entire Site

The Password Protected plugin will do essentially the same thing as option #2 if you’re not comfortable blocking access to the root through the control panel. Do note, however, that this password protection only blocks people from accessing your pages. This plugin will not prevent anyone (including Google) from getting ahold of links to your images or other uploaded files.

Option 5: Use a Plugin to Block Specific Pages

Most SEO plugins enable users to exclude specific pages and posts from being indexed. If you use the SmartCrawl plugin, you can exclude more than that as well, including custom post types, categories, tags, images, stylesheets, and more.

Option 6: Remove from Search After Indexing

Okay, so let’s say your site was mistakenly indexed too early or that you want to remove an old and outdated page from ever showing in search results again. The options above will not give you the ability to remove that already-indexed page from search; you can only block future indexing.

For this option, you’ll need to use your Google Search Console or the corresponding search console tools for other search engines directing traffic to your site.

For Google, here is what you’ll need to do:

  • Go to Search Console and find your site.
  • Click on Google Index and select the “Remove URLs” option.
  • Click on “Temporarily hide” and enter your site’s URL or the URL for the specific page or post you want to remove from search. Make sure the URL you have entered is correct as this will effectively remove your site from search for 90 days.

Take heed of Google’s warning and also note their steps for permanently blocking a site or page from search (as already) noted above).

Wrapping Up

As you can see, it’s a pretty simple process to block Google from indexing your site, which means it’s just as easy to reverse the process (unless you go with option #6). That said, please do remember to reverse the process if this is only meant to be a temporary block!

Obviously, something will seem “off” if your newly completed site goes live and zero traffic shows up, but it shouldn’t get to that point. If blocking search becomes part of your development process, then remember to add a step to remove it as well.

Brenda Barron
Over to you: Have you ever encountered any other reason to block your site or specific content on your site from being indexed by search engines?