SmartCrawl Meta Robots issue

Smartcrawl is reporting that robots.txt is set to forbid crawls on site, and it adds a meta tags to the header

<!-- SEO meta tags powered by SmartCrawl https://premium.wpmudev.org/project/smartcrawl-wordpress-seo/ -->
<meta name="robots" content="noindex,follow" />

  • viobru
    • Support

    Hi, James!

    Hope you are doing great :slight_smile:

    I was checking this and seems that I can't replicate the issue on my testing site.
    Just to let you know, I checked this on 2 different sites and it is working fine to me on both when having SmartCrawl and the WPMU DEV Dashbaord plugins active and running the Twenty Seventen theme on WP 5.0.

    If I check the Search Engine Visibility to "Discourage search engines from indexing this site" under Reading > Settings, the Meta Robots warning is visible, as it should, but as soon as I uncheck it and I run a new Checkup, the Meta Robots warning disappears and I get this: "Your meta robots instructions are currently set to allow search engine bots to crawl and index your page."
    In any case, I get no meta tag added to header.php at any moment.

    Since this might be related to some setting on your end, could you please share the following details so we can have a closer look at this? If this is a live site and no tests can be run on it, please create a staging site on the same server and share its details instead.

    Please use our secure contact form to share them and include the following details on it (DO NOT POST THEM HERE since this is a public thread):

    Subject: "Attn: Violeta" (please use this subject line to ensure that it gets assigned to me :slight_smile: )
    - WordPress admin username
    - WordPress admin password
    - Login URL
    - cPanel credentials (username/password)
    - FTP credentials (host/username/password)
    - Linkback to this thread for reference
    - Any other relevant urls

    Many thanks in advance!

    Kind regards,
    Violeta

  • James
    • New Recruit

    Viobru,

    As I explained to the tech support yesterday I was able to resolve the issue by uninstalling SmartCrawl and then reinstalling it.

    I believe the issue stemmed from using the Yoast Import feature in SmartCrawl which somehow forced a meta tag to display in the header noindex,follow kinda like when you have the checkbox checked for discourage search engines from indexing this site option under reading in settings.

    I appreciate you looking into this but, this is a client site which is live so no way I can run test on that site nor clone it without their permission.

    All I can chalk this up as is a strange occurrence due to the type of configuration that is being used on the site.

    This is the first time ever seeing this plugin do what it was doing yesterday and it defiantly was SmartCrawl because soon as SmartCrawl was disabled that weird forced meta tag added to the header would disappear.

    Checking and unchecking the discourage search engines from indexing this site had no effect in removing this strange forced meta tag either so yeah very strange.

    Either way, as I mentioned I was able to resolve the issue by uninstalling SmartCrawl then reinstalling it, then installing Yoast so I could reimport the SEO metadata but on the second attempt to import I unchecked the box i=n SmartCrawl import settings for import plugin settings from Yoast and everything was fixed and running correctly afterward.

    Thanks again,
    James

  • Patrick Freitas
    • Staff

    Hi James

    Hope you are doing well.

    I'm glad to hear that the plugin is working fine now on your end.

    I tried to replicate importing some settings from Yoast but couldn't replicate this on my end too, seems was a gitch. However, we will keep eyes open on the import settings.

    Let us know if you need any further help on this.
    Best Regards,
    Patrick Freitas

  • samjovanou
    • Flash Drive

    I can confirm I have come across this problem on multiple sites running smartcrawl when using the yoast import feature after deleting yoast.

    As in I think if you “import from yoast” but there is no yoast data to bring across it causes this problem.

    It usually takes about a week to show up in google data console though.

    Had this happen in January to a client which essentially removed their 100+ well-indexed pages with high impressions silently from google and spent a couple weeks trying to figure it out; only solution was switching back to yoast (reinstalling smartcrawl didn’t fix the issue)

    I’ll recreate the error today on one of my websites and see if I can fix it by reinstalling smartcrawl cleanly. Is there any debugging I can do to help fix this possible bug?

  • samjovanou
    • Flash Drive

    Alrighty after a bit of debugging i found that under:
    "SmartCrawl" (in the admin bar on the left)
    --> "Title & Meta"
    "NoIndex" is On

    Troubleshooting done in realtime using search console's live URL tester to confirm first that smartcrawl being enabled/disabled was causing the problem.

    Then after flicking "NoIndex" to off and trying again to import Yoast the problem does not arise again which is good. Still not 100% sure what caused it but was able to recreate it on two different websites (fresh-ish installs with no past SEO plugin data) after installing Smartcrawl and using the import options

Thank NAME, for their help.

Let NAME know exactly why they deserved these points.

Gift a custom amount of points.