[SmartCrawl Pro] Smart Crawl Issue - Crawls won't complete

Also, url crawls get stuck at 13% and then timeout.
https://racquetnetwork.com/wp-admin/admin.php?page=wds_sitemap#tab_url_crawler

I spoke to support earlier. They thought it might be a max execution time issue, but it's not. See here: https://racquetnetwork.com/phpinfo.php

This is also happening on our dev site (dev.racquetnetwork.com). Right now it is stuck at 15%.

When it times out (eventually) I will turn off everything except Smart Crawl and try again. I will update this thread at that time.

  • Majid
    • Recruit

    Hello brentjohner
    I hope you are doing great today :slight_smile:

    Would you please proceed with the plugin conflict test by deactivating all the other plugins except SmartCrawl and WPMU DEV Dashboard plugin (To keep the support access enabled), I did reset the scan from our end so you should be good to run a new one.

    Let us know in a reply the results. Thanks,
    Majid

  • brentjohner
    • The Incredible Code Injector

    Unfortunately, I can't do the plugin test on the live site and I cannot migrate a new copy of the site to our dev site because All in One WP Migration is not working. It will not export a copy of our site. As a result, I cannot do a plugin test.

  • brentjohner
    • The Incredible Code Injector

    OK. So this is frustrating.

    Servmask is trying to figure out why All in One WP Migration isn't working. But I though I would run some tests on the dev site (dev.racquetnetwork.com) while I was waiting.

    1. I turned off everything except Smart Crawl. I ran a new crawl. It seemed to work.

    2. I turned on Woocommerce and all Woocommerce plugins and ran a second crawl. That seemed to work, too.

    3. I turned on Defender Pro. I tried to run another crawl, but it would not run. So I turned it off and tried again. Nothing.

    4. I went back to Step 1 and tried to run a crawl with only Smart Crawl and WPMU DEV Dashboard turned on. Nothing.

    (5. I turned off WPMU DEV Dashboard. That greyed everything out.)

    So now on the DEV SITE I cannot start a new crawl no matter what I do.

    So I am at an impasse.

  • Dimitris
    • Support Star

    Hello there brentjohner,

    hope you're doing good today and please excuse our late response here.

    I couldn't locate anything wrong in the debug.log file and as you've found out, this sounds more like a conflict... Despite of me finding the WP admin and FTP credentials for https://dev.racquetnetwork.com, it seems that the site isn't accessible now.

    Could you please sort this out and let us know here, so we could have a better look in the dev site?

    Warm regards,
    Dimitris

  • Dimitris
    • Support Star

    Hello there brentjohner,

    hope you're doing good today! :slight_smile:

    I just tried to run couple of SEO Checkups in your dev site and everything worked without any issues.
    First, I run one only with WPMUDEV Dashboard, SmartCrawl and WooCommerce plugins.
    Then, I've activated Defender Pro and finally all plugins. In all three cases, SEO Checkup was able to complete.

    Please keep in mind that I didn't wait in the SEO Checkup screen, but I rather chose to close it after initializing the checkup and came back couple of minutes later to check the results.
    Could this be the reasoning of the timeout in live site?
    Can you try to start the SEO Checkup, then navigate away from this screen?
    Let us know, if that made any difference.

    Warm regards,
    Dimitris

  • brentjohner
    • The Incredible Code Injector

    So I have it set to autocrawl at 2:00 am every night. These are the error messages from last night's crawl.

    - Processing timed out.
    - Your SEO crawl has timed out due to an known error. Please try again or contact support.
    - Your SEO crawl has timed out due to an unknown error. Please try again or contact support.
    - The crawl timed out

    As for trying a new crawl as instructed above, that is not working. The NEW CRAWL button on this page is not working. I click it -- nothing happens.

  • Dimitris
    • Support Star

    Hello there brentjohner,

    hope you're doing good today! :slight_smile:

    I accessed dev site again and noticed that last checkup has been made at March 11, 2018 5:26:12 AM and schedule is set daily at 7:00 AM.

    I also tried to run a new scan and the results were the same as my previous reply (now latest scan made at March 11, 2018 5:31:01 AM).

    Were you making above tests in dev (staging) site or in live site? Please advise!

    Warm regards,
    Dimitris

  • brentjohner
    • The Incredible Code Injector

    Hi Dimitris.

    The dev site seems to be working. My concern is with the live site.

    A scan was supposed to run last night, but I can see that it did not complete. It generated the same error messages.

    Also on the live site, when I click on the NEW CRAWL button, nothing happens.

  • Dimitris
    • Support Star

    As I really don't prefer to troubleshoot in the live site, can you please ensure that the dev site is a complete and exact copy of live site?
    You should check that you have the exact same set of plugins, theme and of course settings across whole installation.

    Warm regards,
    Dimitris

  • Dimitris
    • Support Star

    Hey there! :slight_smile:

    I just accessed your live site and I noticed that there's no schedule made. Please activate it and check how's that going.

    I've also tried to run a SEO checkup and this was successful (and pretty fast have to say!).
    Could you please give it another try and let us know how that goes?

    Warm regards,
    Dimitris

  • brentjohner
    • The Incredible Code Injector

    I don't know if this thing is working now or not.

    These are the error messages I am seeing:
    - Processing timed out.
    - Your SEO crawl has timed out due to an known error. Please try again or contact support.
    - Your SEO crawl has timed out due to an unknown error. Please try again or contact support.
    - The crawl timed out

    I also says last crawl March 18, which is yesterday.

    I don't know if that means it completed the crawl or it timed out.

  • Dimitris
    • Support Star

    Hi there! :slight_smile:

    I had another look in your live site and I can see that last SEO Checkup has been made today.

    I've also noticed that SEO Checkup page was being loaded for more that a minute though!!!
    There are some calls that never returns anything, so I'm wondering where did they come from...

    Could you please grant support access for dev site and let us know here in your next reply?
    We could probably narrow this down there, if it's also appearing.

    Warm regards,
    Dimitris

  • Dimitris
    • Support Star

    Hello there! :slight_smile:

    That's true, I can't see any slow queries in the dev site, so there must be some difference(s) with live site.
    Please make sure that dev site is a complete and exact copy of live site, so we could have a better look.
    If you're going to use WP Staging plugin, then you can find more usage details here:
    https://wordpress.org/plugins/wp-staging/

    Warm regards,
    Dimitris

    PS. Please keep in mind that staging environment creations are outside the scope of our support, as they need some more time from our behalf. https://premium.wpmudev.org/docs/getting-started/getting-support/#chapter-1

Thank NAME, for their help.

Let NAME know exactly why they deserved these points.

Gift a custom amount of points.