[SmartCrawl Pro] Posts and pages gradually getting no-index

I noticed in the Google Search Console, that for the last months more and more posts and pages on the website are getting no-index setting. SmartCrawl is globally set to index all pages and posts and I never override that on any page or post I publish, but still when I check some of them, they have the Index – Override Post Type Setting disabled. And it's like that on ~500 out of ~1000 (31.07. = 349, 15.08. = 545), so cannot enable that manually. Any idea why has this happens and how to fix it?

  • Nithin
    • Support Wizard

    Hi Asen Velichkov,

    but still when I check some of them, they have the Index – Override Post Type Setting disabled. And it’s like that on ~500 out of ~1000 (31.07. = 349, 15.08. = 545), so cannot enable that manually. Any idea why has this happens and how to fix it?

    That’s odd behaviour. I checked the SmartCrawl setting, and it does seem to be globally enabled to follow, and index. As you have mentioned, unless the page or post is overridden manually for each it shouldn’t get listed as no-index.

    I checked the random pages, and post in your website but wasn’t able to notice any such tags added in the source code. I have marked this ticket as private, could you please share the links to a couple of pages, and post which Google Search Console is pointing out as having an issue so that we could have a better idea?

    Please do list out the pages so that we could check further, and get this sorted asap.

    Kind Regards,

    Nithin

  • Asen Velichkov
    • Flash Drive

    Hi,

    below is an extract of the links, marked with noindex according to Google Search Console. I copied only some pages and posts, the full list consists of 538 items. Some of them do not need to be indexed, like events, locations, etc, but all the posts and pages should be. And I checked those in the list, they are with index enabled setting from the smart crawl advanced options within each post.

    *After posting this I noticed that the posts, which have url’s in Cyrilic, are not properly encoded here, so maybe ignore those.

    https://vaklush.org/razpileni-biseri/

    https://vaklush.org/sedemte-lacha/

    https://vaklush.org/?????-??-????????/

    https://vaklush.org/spisanie-nur/

    https://vaklush.org/duhovnite-darove-2/

    https://vaklush.org/??????-?-????????????-?????-??-??????/

    https://vaklush.org/??????-???-?-?????-????-?-???????/

  • Asen Velichkov
    • Flash Drive

    Meanwhile I noticed, that Google Search Console returns a parsing error, trying to read the website sitemap. When I submit it, it says success, but after a while it says:

    “Sitemap can be read, but has errors

    Parsing error

    1 instance

    We were unable to read your Sitemap. It may contain an entry we are unable to recognize. Please validate your Sitemap before resubmitting.

    Examples

    Line 1″

  • Nithin
    • Support Wizard

    Hi Asen Velichkov,

    I checked these pages, and I don’t see any instances where these mentioned post/pages are marked as no-index in general when checking the page source.

    I could notice the posts listed in the previous replies are listed in the sitemap.xml file of your website, if the posts were marked as no-index via SmartCrawl it wouldn’t have listed in the sitemap.xml file too.

    I’m afraid, there aren’t any instances we could detect which is causing within SmartCrawl plugin side. Do you have server-side cache enabled? Could you please clear the cache, and then check whether it makes any difference?

    I also checked the Sitemap via online tools, but I’m not able to notice any such issues as stated:

    https://www.websiteplanet.com/webtools/sitemap-validator/

    https://seomator.com/free-tools/sitemap-test

    Maybe you could try disabling the Style Sitemap option, under SmartCrawl > Sitemap > Settings, and then see whether you still notice issues with Sitemap.

    Please do double-check with your hosting provider regarding server cache, and once it’s cleared please re-crawl the site again, and see whether it helps.

    If you still have issues even after that then please create a staging, and check how does Google Search Console behave when the staging site is added as a new property.

    Kind Regards,

    Nithin

Thank NAME, for their help.

Let NAME know exactly why they deserved these points.

Gift a custom amount of points.