SmartCrawl causes a publishing slowdown when a large database is present

We are having an issue with SmartCrawl. It is causing the publishing of posts to stall. When SmartCrawl is active, publishing a post results in WP returning the Connection Lost error. Generally, when SmartCrawl is active, it takes WP about 2 minutes or so to publish a post

I've been working on a staging site to test it through and the site has been scaled down to only a default theme and SmartCrawl, with AutoLinking turned off. The only complicating factor remaining on this site currently is the large database. It has 20,000 standard posts and 8,000 additional posts of a custom post type (which not being used since the plugin creating them is deactivated as part of this testing.)

My suspicion is that SmartCrawl doesn't play well with large databases, which could be considered a major problem for an SEO plugin.

The site is located here:

It is a staging site and it is HTML auth blocked. Login with user: flywheel and pass: current-lift

Support access is activated in WPMU DEV Dashboard. Feel free to login and test as you need too to confirm my findings, but I suspect that large databases might be a weakness for this plugin.

  • James Morris

    Hello Saunt Valerian,

    I hope you are well today.

    I've taken a look at your site and found a few issues that will cause the behavior you're seeing.

    First, SmartCrawl was set to automatically update the sitemap on each post/page publication. While this will work just fine for most websites, site's like yours are not the average. Such a large database of entries takes a little time to regenerate in a sitemap. If you go to WP Admin -> SmartCrawl -> Sitemap -> Options and turn on Disable Automatic Sitemap Updates, this will prevent SmartCrawl from trying to update the sitemap on every post/page publication or update.

    NOTE: After disabling automatic sitemap updates, I created a test post and it published within a couple seconds.

    Another issue I found was when I checked your PHP and WordPress configuration.

    PHP Config:

    max_execution_time = 90
    memory_limit = 128M

    WP Config:

    You can see these values at WP Admin -> WPMU DEV Dashboard -> Support -> System Info.

    For a site of this size, I'd recommend no less than 512M for memory and at least 180 for execution time. You're doing a lot a data handling and the more memory you have to work with, the faster this data can be processed.

    To learn how to increase your memory and execution time, please see the following links:

    Let us know if you have any further issues. We'll be happy to help! :slight_smile:

    Best regards,

    James Morris

Thank NAME, for their help.

Let NAME know exactly why they deserved these points.

Gift a custom amount of points.