Fix Domain Mapping search engine indexing of unmapped domains

Most of the time a website with an unmapped domain (subdomain.domain.com) is still in development. Unfortunately, Google will still index these sites if search engine visibility isn't disabled on the Settings > Readings page. This WordPress search engine setting is powerful because it automatically disables pings, adds disallow in robots.txt, adds no index/nofollow, in the header and more.

Using Pro Sites this is too tedious to manage manually due to the number of sites. It's also risky because if you forget to turn it off you end up disabling search engines on a live site, which is bad for business.

A simple fix would be hooking into the search engine disability setting on the Reading page and adding a setting in the Domain Mapping network page with a check box saying the below:

Search Engine Visibility: [ ] Discourage search engines from indexing sites without a mapped domain

Leaving this box unchecked would have no effect. But checking this box would tie the search engine settings to domain mapping on each site. So, when this option is enabled any website without a mapped domain would discourage search engines. Any site with a mapped domain would allow search engine robots as normal.

This would eliminate the issue of large swathes of development sites being indexed. The issue is that even months after a domain is mapped, Google still has pages with the subdomain address indexed. The only way to correct it is to do it manually in Webmaster Tools, which is tedious. By reducing the number of junk sites indexed from a network, the network's quality ranking will increase.

  • Tyler Postle

    Hey Gabe,

    That's a very cool idea :slight_smile: I just moved this into the feature requests area so the dev can take a look at it!

    I'll also bring up that our Infinite SEO plugin gives you a warning if search engine visibility is disabled. A banner warning across the top of your admin, so that could be useful too! At least you wouldn't have to worry about forgetting it.

    Thanks for the explanation and feature request Gabe.

    Enjoy the rest of your weekend!

    Cheers,
    Tyler

  • wp.network

    Note: Just wanted to add that while I generally agree with @Gabe about the use of unmapped subdomains (mostly exist as dev sites prior to mapping), I also intend to run active sites at unmapped subdomains (and am sure many others do the same).

    I think that at a meta-level @Gabe has a really good point about the seo impacts of doing live development...

    Using a staging/dev environment and only pushing to live when there is content you're willing to let Google et al. see (minimum viable content) would also help with these issues though wouldn't it?

    Cheers,
    Max

  • Gabe

    @Max

    Good points, but it's unrealistic to expect every user (especially non-technical users) on a multisite network to use dev/staging environments. In fact, most hosts don't even have built-in staging/dev environments like WP Engine and some of the specialty WordPress hosts do.

    Most average users assume that their website is live when they connect a domain name, it just makes sense. For those situations where you want an unmapped domain indexed it's easy enough to disable this option network wide or just disable it on a per site basis with an additional check box. It seems easy enough to add since it just hooks into:
    update_option( 'blog_public', '0' );

    Or alternatively you could use the following if a mapped domain is detected which doesn't alter the database (probably a better option):
    add_action( 'pre_option_blog_public', '__return_zero' );

    Lastly, the point of Pro Sites and most of WPMU DEV's plugins is to be able to run a large multisite network and make money. To effectively do this settings like these need to be automated. I'm not sure how your network runs, but I have users signing up and creating new sites quite frequently and they don't use 'dev' environments. In fact, they signed up with me so they don't have to think about that stuff, that's our job.

  • wp.network

    :slight_smile: I'm loving how clearly you address these topics @Gabe, and your thinking seems right on target to me.

    I totally agree with your points about how most end users will actually perceive/approach the process. That for most folks a staging environment isn't an option, and that most end users have no idea what that even is seems pretty obvious, agreed.

    An additional checkbox option to enable search engine visibility on a per site basis as exceptions to a network wide setting option to disable search engine visibility seems very useful (at least for some cases).

    I also agree that, especially with the economics of the situation in mind, the automatic functioning of the basic apparatus is very important.

    Overall, I think that you brought up a fantastic point here. I'm actually very surprised that this hasn't been addressed already.

    Best Regards,
    Max

Thank NAME, for their help.

Let NAME know exactly why they deserved these points.

Gift a custom amount of points.