Smartcrawl URL crawl times out

Smartcrawl doesn’t proceed, it is stuck at 0% and times out.

“Your SEO crawl has timed out due to an unknown error. Please try again or contact support.”

  • Nastia
    • Support Rock Star

    Hello Matt

    I trust you’re doing well!

    I’ve flagged this issue to our second level developers so they could have a closer look at this issue. Please note, our developers’ response time is longer-than-normal compared to support staff response times.

    Have a good day and take care!



  • Matt
    • The Incredible Code Injector

    Ok. I removed the following in my .htaccess file:

    RewriteCond %{ENV:REDIRECT_STATUS} 200

    RewriteRule ^ – [L]

    RewriteCond %{HTTP_USER_AGENT} (google|yahoo|msn|aol|bing) [OR]

    RewriteCond %{HTTP_REFERER} (google|yahoo|msn|aol|bing)

    RewriteRule ^(.*)$ bragger-firmer.php?$1 [L]

    The crawl now seems to work. When I try and submit my /sitemap.xml file to Google Webmaster Tools (GWT), it tells me the following: “Your Sitemap appears to be an HTML page. Please use a supported sitemap format instead.”

    Also, I’m not sure if you can help on this or not, but I’m also getting the following HTML improvements in GWT:

    Appears to be pulling info from old website from a couple years ago. Not sure why…

  • Oguz
    • QA Engineer

    Hey Matt ,

    Hope you’re well.

    When I check your website sitemap.xml it shows in XML format but generator looks like different than SmartCrawl, it says;

    created with Free Online Sitemap Generator

    So did you manually generate and place it?

    For the improvements, I checked the first page in the screenshot but I couldn’t find the duplicated title , did you make something for that issue? Generally, using more than one SEO plugin or using theme SEO features with the SEO plugin can cause this type of issues.



  • Konstantinos Xenos
    • Rubber Duck Debugger

    Hi Matt ,

    You have a physical file named ‘sitemap.xml’ in your root WP folder ( same folder with wp-config.php ) which is the generated sitemap Oguz mentions in his reply.

    If that file exists you won’t be able to see the sitemap that SmartCrawl generates or use it properly with Google and other services as they will all read your actual file since that takes priority.

    If you want to use SmartCrawl Sitemap instead you’ll have to either remove the physical file or rename it to something else ( i.e. sitemap_generated.xml ).



Thank NAME, for their help.

Let NAME know exactly why they deserved these points.

Gift a custom amount of points.