[SmartCrawl Pro] Too many server errors


When I want to do a new crawl on my multisite, I'm getting these errors:
Too many server errors + Your SEO crawl has timed out due to an unknown error. Please try again or contact support.

What would be the solution?


  • Adam Czajczyk
    • Support Gorilla

    Hello Igepa2R

    I hope you're well today and thank you for your question!

    When the crawl is started it actually crawls all the pages/links of your site. When it encounters e.g. link that's responding with 404 Not Found (so a link to non-existing content) it's counted as an error. If the site responds with any of the 50x errors (500, 502, 503 and so on - all basically related, more or less, to gateway timeouts/server resources), it also counts as an error. if there's too many errors the crawl is aborted because it doesn't make much sense to continue it - you don't want many 404 and other "faulty" URLs in your sitemap.

    When you look at the crawl page in the plugin there should be more information about where the errors were encountered so those URLs should be checked and, if possible, the steps should be taken to fix these errors there then run crawl again. If there are more errors detected again, this should be done as many times as necessary until the crawl is able to complete.

    However, these errors might also be caused by some other reasons such as e.g. conflict with some other plugin, some non-standard configuration specific to this particular site or because server is simply not-able to handle that amount of requests for the site that is generated by the crawl.

    That said, would you mind enabling support access to the site so I could take a closer look at it? To do that, please go to the "WPMU DEV -> Support" page in your site's back-end, click on "Grant support access" button there and let me know here once it's done.

    I'll check the site then and we'll decide what to do next to solve that.

    Best regards,

Thank NAME, for their help.

Let NAME know exactly why they deserved these points.

Gift a custom amount of points.