SmartCrawl Sitemap indexing

Hi! I have an issue with SmartCrawl where it is not indexing all my URL's. I have about 16000 products on my page but only 1000 URL's are discovered.
Error: "Your SEO crawl has timed out due to an unknown error. Please try again or contact support."
Support access granted on my site.

  • Adam Czajczyk
    • Support Gorilla

    Hello Sprenzy

    I hope you're well today and thank you for your question!

    The site URL crawl is very "resource intensive" and because of that (in order to prevent server resources) there's a default limit of 1000 URLs to be scanned. You can increase that limit by adding a line to the "wp-config.php" file:

    define( 'WDS_SITEMAP_POST_LIMIT' , 20000 );

    The example above would increase the limit to 20000 URLs but you may set it to any other number that would suffice for your site. It might also be good to raise WP memory a bit more in this case. Currently it's set to allow up to 256M of memory to be used but with increased crawl limits I'd suggest going for e.g. 512M.

    You should already have this line in the "wp-config.php" file:

    define( 'WP_MEMORY_LIMIT', '256M');

    so just change 256M to 512M there.

    Make sure also that both these lines are above the "/* That's all, stop editing! */" line.

    Give it a try please and let me know if it helped.

    Best regards,
    Adam

  • Sprenzy
    • Design Lord, Child of Thor

    Hi Adam,

    Thank you for the fast reply!

    I have done as suggested and it seems as though the sitemap is now correct. In the WP dashboard it shows 16000+ links indexed on the SmartCrawl widget. The .xml file submitted on google webmaster tools shows the same amount of URL's.

    The SmartCrawl > Sitemap settings page however still shows 1000 URL's discovered and takes a rather long time to load. Possibly something to fix.

    Thanks for the assist!

    Regards,
    Alex

Thank NAME, for their help.

Let NAME know exactly why they deserved these points.

Gift a custom amount of points.