Smartcrawl using a lot of server's resources

At every page load a crawl is being done by the plugin with the following function:
Smartcrawl_OnPage->smartcrawl_head() | wpmu-dev-seo/includes/tools/onpage.php:303 | 10.2949s

The page load time has increased to 15-20seconds. Server support followed up with the following:
We see in the hook_time that the plugin is eating 16s in the example below:
+------------+----------+------------+-------------+-------------+------------+--------------+-----------+------------+--------------+---------------+
| stage | time | query_time | query_count | cache_ratio | cache_hits | cache_misses | hook_time | hook_count | request_time | request_count |
+------------+----------+------------+-------------+-------------+------------+--------------+-----------+------------+--------------+---------------+
| bootstrap | 0.5683s | 0.0132s | 64 | 86.91% | 664 | 100 | 0.1988s | 5465 | 0s | 0 |
| main_query | 0.9699s | 0.0047s | 8 | 84% | 105 | 20 | 0.9673s | 58071 | 0s | 0 |
| template | 16.821s | 0.117s | 132 | 99.4% | 31199 | 188 | 16.797s | 210504 | 0s | 0 |
+------------+----------+------------+-------------+-------------+------------+--------------+-----------+------------+--------------+---------------+
| total (3) | 18.3592s | 0.1349s | 204 | 90.1% | 31968 | 308 | 17.9631s | 274040 | 0s | 0 |
+------------+----------+------------+-------------+-------------+------------+--------------+-----------+------------+--------------+---------------+

The SmartCrawll is bypassing caching completely, and it cannot serve out of cache the result is 100 CPU usage and a lot of RAM usage.

  • Adam Czajczyk

    Hello Azer Guenes

    I hope you're well today and thank you for your question!

    The "smartcrawl_head()" function referenced in your post is essential as it is printing out all the "SEO tags" (meta, og, twitter cards, canonical links etc) into the site header. The function itself is pretty simple and shouldn't be causing such a load in any way.

    However, it's also reading some options first (it has to do it in order to know what to print out) and I assume that this might be the culprit here: if there are tons of SmartCrawl related data in the database (e.g. due to some malfunction of a site or a plugin or just database) or the database _options table is corrupted or overloaded (it shouldn't be too big) - this seems like something that could slow that down, thus causing such issue.

    That said, I'd like to include our developers in that in order to diagnose the case precisely but they'll need to be able to access the site directly: to check files and the database. For this, we'll need full access credentials so would you please provide them?

    Note: Don't leave your login details in this ticket.
    Instead, you can send us your details using our contact form https://premium.wpmudev.org/contact/#i-have-a-different-question and the template below:

    Subject: "Attn: Adam Czajczyk"
    - Site login URL
    - WordPress admin username
    - WordPress admin password
    - FTP credentials (host/username/password)
    - cPanel credentials (host/username/password)
    - Folder path to site in question
    - Link back to this thread for reference
    - Any other relevant urls/info

    Best regards,
    Adam

Thank NAME, for their help.

Let NAME know exactly why they deserved these points.

Gift a custom amount of points.