SmartCrawl URL crawler - Total URLs Discovered

SmartCrawl URL crawler shows that Total URLs Discovered is 314, but I have over 1000 URLs on my site. I see that in sitemap I have 1074 URLs.
Why is the Total URLs Discovered still 314?

  • Kasia Swiderska
    • Support nomad

    Hello Peter,

    I spoke with developers and they said that SEO Crawl goes through the site, parses links and discovers URLs in that way. It discovers linked content. And these can, but don't have to be, represented in the sitemap. So different result between the Sitemap and Total URLs Discovered is not a bug.

    Hi Kasia, don't forget the message I'm getting on SmartCrawl that says 'Your meta robots instructions are currently set to forbid search engine bots to index your page.'

    Where exactly do you see this message? I'm checking your site now and I don't see anything like that. I also checked your site's robotx.txt and it doesn't look like it blocking all the crawlers.

    kind regards,
    Kasia

  • Peter
    • New Recruit

    Hi Kasia

    Sorry for the late reply, I didn't know you'd posted a response.

    Thanks for checking out the difference between the two results, that makes sense. I've also just checked the SEO report and the message about the Robots.txt has gone so hopefully it was just a glitch.

    Thank you for all your help, it's much appreciated.

Thank NAME, for their help.

Let NAME know exactly why they deserved these points.

Gift a custom amount of points.