Smart Crawl - Error Crawling

There was an error crawling we weren't able to get meaningful results

  • Adam Czajczyk
    • Support Gorilla

    Hello Wendy

    I hope you're well today and thank you for your question!

    I can confirm that in our logs SEO crawl is reported as started and immediately terminated and I also got the same error when trying to crawl your site from our end.

    That usually means that there's "something" that's breaking/blocking our crawlers (SmarCrawl plugin initiates scan but the scan itself is run from our servers). That said, do you have any additional firewall/security/caching plugins installed on site that could potentially reject the crawler access?

    I would also like to take a closer look at your site's configuration so could you please enable support access for me? To do this, please go to the "WPMU DEV -> Support" page in your site's back-end and click on "Grant Support Access" button there, then let me know here.

    Best regards,
    Adam

  • Predrag Dubajic
    • Support

    Hi Wendy,

    Apologies for the delay here.

    I had a look at your site and unfortunately, there's information in console or network logs during the scan, it doesn't show any errors that could point us in the right direction.

    Can you enable debug log in your wp-config.php file (located in root WP folder) by replacing define('WP_DEBUG', false); with this code:

    // Enable WP_DEBUG mode
    define('WP_DEBUG', true);
    
    // Enable Debug logging to the /wp-content/debug.log file
    define('WP_DEBUG_LOG', true);
    
    // Disable display of errors and warnings
    define('WP_DEBUG_DISPLAY', false);
    @ini_set( 'display_errors', 0 );

    Run the new scan and once it fails it should create debug.log file in your wp-content folder that could tell us what is going on.
    Can you please save it as .txt file and attach it here so we can check it out.

    Also, could you ask your hosting provider to check their server logs and see if there's something blocking outside requests to your site?

    Best regards,
    Predrag

  • Predrag Dubajic
    • Support

    Hi Wendy,

    Please make sure that you have renamed debug.log to debug.txt as .log extension is not allowed attachment.
    Also, if your log file is over 5MB it could be due to that, but if you enabled debug log just not it shouldn't be that large.
    If it is however, then please update the file to any file sharing service like Dropbox or google drive and share the download link here.

    Best regards,
    Predrag

  • Predrag Dubajic
    • Support

    Hi Wendy,

    Hmm, this is really strange, there's only one line there and it doesn't tell us much I'm afraid.

    Is it possible to provide us with FTP or cPanel login details so we can investigate this further?
    You can send us your details using our contact form https://premium.wpmudev.org/contact/#i-have-a-different-question and the template below:

    Subject: "Attn: Predrag Dubajic"
    - Site login URL
    - WordPress admin username
    - WordPress admin password
    - FTP credentials (host/username/password)
    - cPanel credentials (host/username/password)
    - Link back to this thread for reference
    - Any other relevant urls/info

    Best regards,
    Predrag

  • Adam Czajczyk
    • Support Gorilla

    Hello Wendy

    I have just checked our internal mailing system and I didn't find any message from you. It rarely happens but it's possible that for some reason it got into some spam filters on a way.

    Would you please try to send again? Please follow the format requested by my colleague Predrag but this time instead of using our contact form, just send a regular e-mail directly to "contact@wpmudev.org" and if you use specifically the same subject line, it should come through and be assigned to Predrag.

    I apologize for inconvenience and thank you for your help!

    Best regards,
    Adam

  • Wendy
    • WPMU DEV Initiate

    Hi, I don't know if anything else can be done to resolve this :slight_frown: I have contacted my web host and we have white listed Wpmudev ip address. They can't find anything blocking either. Still not working :weary:

  • Predrag Dubajic
    • Support

    Hi Wendy,

    I'm afraid that the issue here is not just SmartCrawl scan, the thing is that your site is having general accessibility issues, at the moment I can't even access your site from my location and I need to use VPN in order to open it.

    Did you get any info from your host about why your site is not being accessible from the different locations?

    Best regards,
    Predrag

Thank NAME, for their help.

Let NAME know exactly why they deserved these points.

Gift a custom amount of points.