[SmartCrawl] Crawler not working

Tried to crawl the site but getting error:
“We weren’t getting meaningful results”

  • Nithin
    • Support Wizard

    Hi WP HOUS – WordPress Specialist ,

    I checked your website and I could notice the issue. Could you please enable debug mode in your website so that we could troubleshoot the issue further.

    To enable debug mode, open your wp-config.php file located in your root directory, and look for define(‘WP_DEBUG’, false);. Change it to:

    define('WP_DEBUG', true);

    In order to enable the error logging to a file on the server you need to add:

    define( 'WP_DEBUG_LOG', true );
    define( 'WP_DEBUG_DISPLAY', false );
    @ini_set( 'display_errors', 0 );
    define( 'SCRIPT_DEBUG', true );
    define('SMARTCRAWL_ENABLE_LOGGING', true);

    After making the above changes, please try run a new SEO Checkup. If the checkup fails, the related errors will be saved to a debug.log log file inside the /wp-content/ directory, and in wds-seo-log.php inside /wp-content/uploads/ directory.

    Please attach these log files in your next reply in txt format file, so that we could give a closer look. You can also share the files via Google Drive, Dropbox etc if you face any issues with sharing the log files.

    You can find more details about debugging here.

    Please let us know how that goes, have a nice day.

    Kind Regards,

  • Muhammad Sidiq
    • Flash Drive
    <?php die(); ?>
    [2019-10-09 05:37:53][10] Forced timeout on sitemap crawl
    [2019-10-09 05:37:55][10] Sending start request to remote checkup service
    [2019-10-09 05:37:55][10] Sending a remote request to [https://premium.wpmudev.org/api/seo-checkup/v1.1/start?domain=https%3A%2F%2Fnurhampers.com] (start)
    [2019-10-09 05:38:05][50] We had an error communicating with [https://premium.wpmudev.org/api/seo-checkup/v1.1/start?domain=https%3A%2F%2Fnurhampers.com]:[500] (start).
    [2019-10-09 05:38:05][50] Response from remote checkup service has no data
    [2019-10-09 05:38:19][10] Sending a remote request to [https://premium.wpmudev.org/api/seo-checkup/v1.1/result?domain=https%3A%2F%2Fnurhampers.com] (result)
    [2019-10-09 05:38:30][10] Sending a remote request to [https://premium.wpmudev.org/api/seo-checkup/v1.1/emails] (emails)
    [2019-10-09 05:38:30][50] We had an error communicating with [https://premium.wpmudev.org/api/seo-checkup/v1.1/emails]:[500] (emails).
  • Nithin
    • Support Wizard

    Hi Muhammad Sidiq ,

    From the logs given, the issue seems more related to website timing out. If I load the website in incognito mode, the website takes a lot of time to load.

    In Firefox, I could notice it’s throwing Time out errors most of the time:

    I’m afraid, this is something which I would highly recommend checking with your hosting provider, so that that they could help in pinpointing anything specific causing the time out, and the website taking so much time to load.

    Also please try to increase the server resources, and see whether that helps. You’ll have to create a file called php.ini in the root directory, and add the following rules:

    upload_max_filesize = 128M
    post_max_size = 256M
    memory_limit = 512M
    file_uploads = On
    max_execution_time = 800

    The root directory is where wp-config.php file resides. To ensure the values have increased, you can check under WPMU DEV > Support > System Info.

    Please check the following article for more info:

    If the changes done doesn’t reflect in the WPMU DEV Dashboard plugin, then most probably your host doesn’t support increasing the resources via php.ini file. In that case, you’ll have to bring this into your hosting provider’s attention to make the required changes.

    Please let us know how that goes. Have a nice day ahead.


  • Nithin
    • Support Wizard

    Hi Muhammad Sidiq ,

    Sorry for missing out the site was hosted in WPMU DEV. It wasn’t listed in the ticket regarding that and hence got overlooked. SmartCrawl now has a new release of version 2.4.

    So, I took a manual backup of your website in our Host and updated to the latest version to see whether it helps, but I’m afraid the issue still exists.

    On further troubleshooting, I could notice the following logs which might be primarily causing the issue with the website taking time to load:
    [10-Oct-2019 08:44:20 UTC] PHP Warning: session_start(): Cannot start session when headers already sent in /xxxxl/wp-content/plugins/admin/public/class-thwcfe-public-checkout.php on line

    The above logs are related to the “WooCommerce Checkout Field Editor Pro” plugin. I quickly disabled the “WooCommerce Checkout Field Editor Pro” plugin, and the homepage was loading much faster compared to the plugin enabled.

    I tried to run SmartCrawl SEO Checkup with the above-mentioned plugin disabled to see whether it helps, but I’m afraid the issue still exists.

    I have enabled the mentioned plugin back. I’m pinging our developers regarding the anomalies noticed, and see what could be causing the issue further. Will get back to you once we get further feedback regarding the status of the issue asap.


Thank NAME, for their help.

Let NAME know exactly why they deserved these points.

Gift a custom amount of points.