block existing blogs and newly created blogs from all crawlers

Hello,

I have a site with approx. 20 separate blogs. When a new user creates a new blog, the following url is created:

http://www.mysite.com/newblog21/

with “newblog21” being the user login.

I would like to set it up so that all existing blogs and newly created blogs are blocked from all crawlers. Keeping in mind that the home page ('http://www.mysite.com') and pages created from the home site (i.e. "http://www.mysite.com/aboutus") is still crawl-able.

Any direction is appreciated.

  • Ivan

    Hey there grego,

    Hope you are well today.

    I see you are using folders for your network. So we can come up with some very simple solution you can try by editing your robots.txt in the main site. Something like

    User-agent: *
    Disallow: /newblog21/
    Disallow: /newblog42/
    Disallow: /futureblog/

    However this is manual solution and I am not aware of any plugin that will automatically disable future sites in the robots.txt

    You can also take a look at this one - https://wordpress.org/plugins/multisite-robotstxt-manager/
    You can control more complex rules for every blog, but it might be a little overkill for you.

    Let me know if this helps.

    Have a great day,
    Ivan

  • aristath

    Hello there @grego3781, I hope you're well today!

    You can simply enable the "Discourage search engines from indexing this site" option on your dashboard. You can find that option in Settings => Reading.
    If you want all new sites on your network to have that option disabled then you can use our New Blog Templates plugin: https://premium.wpmudev.org/project/new-blog-template/
    Using that, you can create a "template" blog that all new sites will use when they are being created. You can then enable that option on the template site and that's all. :slight_smile:

    If you need to force this option to be on on ALL your sites, then you can simply ignore my previous suggestion on using New Blog Templates and do the following:

    Create a new php file in your wp-content/mu-plugins folder (if that folder does not already exist, manually create it) and paste the following in that file:

    <?php
    
    function custom_force_nonpublic_site() {
    
      $public = get_option( 'blog_public', 1 );
    
      if ( 0 != $public ) {
        update_option( 'blog_public', 0 );
      }
    
    }
    add_action( 'wp', 'custom_force_nonpublic_site' );

    This will check the value of the option I mentioned above, and if the site is public, then forces it to be non-public.

    I hope that helps!

    Cheers,
    Ari.

  • aristath

    Hello again @grego3781, I hope you're well today!

    You can have an array of blogs that you force to be public and the rest will be forced to be non-public. To do that, you will have to use the below code instead of the one I posted above:

    <?php
    
    function custom_force_nonpublic_site() {
    
        // If you want to force multiple sites to be public,
        // you will have to add them in this array.
        // Example: force blogs with ids 1, 3 & 5 to be public:
        // $force_public = array( 1, 3, 5 );
        // Below we're forcing ONLY blog with id=1 to be public
    
        $force_public = array( 1 );
    
        // Get the current blog ID
        $blog_id = get_current_blog_id();
    
        // Get the option from the database
        $public = get_option( 'blog_public', 1 );
    
        // If the current blog ID is in the array of public blogs
        // then force it to be public
        if ( in_array( $blog_id, $force_public ) ) {
    
            if ( 1 != $public ) {
                update_option( 'blog_public', 1 );
            }
    
        // If the current blog ID is NOT in the array of public blogs
        // then force it to be non-public
        } else {
            if ( 0 != $public ) {
                update_option( 'blog_public', 0 );
            }
        }
    
    }
    add_action( 'wp', 'custom_force_nonpublic_site' );

    I hope that helps!

    Cheers,
    Ari.

Thank NAME, for their help.

Let NAME know exactly why they deserved these points.

Gift a custom amount of points.