[SmartCrawl Pro] Not creating sitemap.xml

SmartCrawl error log says:

[2019-05-27 14:37:39][50] We had an error communicating with [https://premium.wpmudev.org/api/seo-checkup/v1.1/emails]:[500] (emails).
[2019-05-27 17:33:20][40] Unable to obtain request arguments for [emails].

1) I cannot perform an SEO Checkout (without timing out). It'll run up to 99%, then kick out/timeout.

2) Every time I build a sitemap.xml, it says 1,000 URLs discovered, but no sitemap.xml is available

Initially, I could not run a sitemap scan. I noticed that the sitemap search problem came from having incorrect "post types" included in my wp_posts table. (oddly, the post "SLUG" was used in place of the post-type "post".) I'm not 100% certain, but I believe that these incorrect entries either came from AutoBlog or a WP All Import, that mismapped some of the terms & fields... somehow.

After I removed these posts types, (replacing the slug w/ "post") I was able to run the sitemap scan. But, even though I can see that URLs have been collected, they're still not entering into the sitemap.xml page (which shows as a blank page).

  • splaquet
    • The Incredible Code Injector

    As a quick update... I submitted that through the site's support access portal, but did not notice any attention to it. I reposted the support request here before noticing that the error log had moved past that initial few lines.

    I saw on someone else's support request that their site wasn't viewable from the web. I knew that my site was, but I figured I'd change the IP address on the server... just to try things out. Not sure how or why, but that was the only thing that i did (I believe) that got SmartCrawl crawling again.

    - It still shows 1,000 URLs found, when I have the wp-config override in for 10,000

    I have these additional lines in my wp-config:
    define( 'SMARTCRAWL_SITEMAP_POST_LIMIT', 10000 );
    define( 'SMARTCRAWL_EXPERIMENTAL_FEATURES_ON', true );
    define( 'SMARTCRAWL_ENABLE_LOGGING', true );
    define('SMARTCRAWL_DEBUG_LOG_LEVEL', 10);
    define( 'SMARTCRAWL_ANALYSIS_REQUEST_TIMEOUT', 25 );

    No matter what I try, I simply cannot seem to have the sitemap.xml generate:
    https://utterbuzz.com/sitemap.xml

    side note, but i found a spelling error in one of the lines in the error log: "cralwer"
    [2019-05-27 23:09:05][10] Setting up cralwer schedule

    I don't want to overwhelm anyone with too much error log info, but here's the copy/paste of the entire error log:

    [2019-05-27 14:37:39][50] We had an error communicating with [https://premium.wpmudev.org/api/seo-checkup/v1.1/emails]:[500] (emails).
    [2019-05-27 17:33:20][40] Unable to obtain request arguments for [emails].
    [2019-05-27 17:33:43][40] Unable to obtain request arguments for [emails].
    [2019-05-27 17:41:31][50] We had an error communicating with [https://premium.wpmudev.org/api/seo-checkup/v1.1/emails]:[500] (emails).
    [2019-05-27 17:55:41][50] We had an error communicating with [https://premium.wpmudev.org/api/seo-checkup/v1.1/emails]:[500] (emails).
    [2019-05-27 18:22:23][50] We had an error communicating with [https://premium.wpmudev.org/api/seo-checkup/v1.1/emails]:[500] (emails).
    [2019-05-27 18:53:59][50] We had an error communicating with [https://premium.wpmudev.org/api/seo-checkup/v1.1/emails]:[500] (emails).
    [2019-05-27 19:01:31][10] Sending start request to remote checkup service
    [2019-05-27 19:01:31][10] Sending a remote request to [https://premium.wpmudev.org/api/seo-checkup/v1.1/start?domain=https%3A%2F%2Futterbuzz.com%2F] (start)
    [2019-05-27 19:01:34][20] (Re)generating sitemap
    [2019-05-27 19:01:42][10] Sending a remote request to [https://premium.wpmudev.org/api/seo-checkup/v1.1/result?domain=https%3A%2F%2Futterbuzz.com%2F] (result)
    [2019-05-27 19:01:51][10] Sending a remote request to [https://premium.wpmudev.org/api/seo-checkup/v1.1/result?domain=https%3A%2F%2Futterbuzz.com%2F] (result)
    [2019-05-27 19:01:59][10] Sending a remote request to [https://premium.wpmudev.org/api/seo-checkup/v1.1/result?domain=https%3A%2F%2Futterbuzz.com%2F] (result)
    [2019-05-27 19:02:08][10] Sending a remote request to [https://premium.wpmudev.org/api/seo-checkup/v1.1/result?domain=https%3A%2F%2Futterbuzz.com%2F] (result)
    [2019-05-27 19:02:16][10] Sending a remote request to [https://premium.wpmudev.org/api/seo-checkup/v1.1/result?domain=https%3A%2F%2Futterbuzz.com%2F] (result)
    [2019-05-27 19:02:25][10] Sending a remote request to [https://premium.wpmudev.org/api/seo-checkup/v1.1/result?domain=https%3A%2F%2Futterbuzz.com%2F] (result)
    [2019-05-27 19:02:33][10] Sending a remote request to [https://premium.wpmudev.org/api/seo-checkup/v1.1/result?domain=https%3A%2F%2Futterbuzz.com%2F] (result)
    [2019-05-27 19:02:41][10] Sending a remote request to [https://premium.wpmudev.org/api/seo-checkup/v1.1/result?domain=https%3A%2F%2Futterbuzz.com%2F] (result)
    [2019-05-27 19:02:50][10] Sending a remote request to [https://premium.wpmudev.org/api/seo-checkup/v1.1/result?domain=https%3A%2F%2Futterbuzz.com%2F] (result)
    [2019-05-27 19:02:58][10] Sending a remote request to [https://premium.wpmudev.org/api/seo-checkup/v1.1/result?domain=https%3A%2F%2Futterbuzz.com%2F] (result)
    [2019-05-27 19:03:12][20] (Re)generating sitemap
    [2019-05-27 19:03:41][20] (Re)generating sitemap
    [2019-05-27 19:05:34][20] (Re)generating sitemap
    [2019-05-27 19:35:04][10] Sending a remote request to [https://premium.wpmudev.org/api/seo-checkup/v1.1/result?domain=https%3A%2F%2Futterbuzz.com%2F] (result)
    [2019-05-27 19:35:05][10] Sending a remote request to [https://premium.wpmudev.org/api/seo-checkup/v1.1/emails] (emails)
    [2019-05-27 19:35:06][50] We had an error communicating with [https://premium.wpmudev.org/api/seo-checkup/v1.1/emails]:[500] (emails).
    [2019-05-27 19:35:08][10] Sending start request to remote checkup service
    [2019-05-27 19:35:08][10] Sending a remote request to [https://premium.wpmudev.org/api/seo-checkup/v1.1/start?domain=https%3A%2F%2Futterbuzz.com%2F] (start)
    [2019-05-27 19:35:18][10] Sending a remote request to [https://premium.wpmudev.org/api/seo-checkup/v1.1/result?domain=https%3A%2F%2Futterbuzz.com%2F] (result)
    [2019-05-27 19:35:19][10] Sending a remote request to [https://premium.wpmudev.org/api/seo-checkup/v1.1/emails] (emails)
    [2019-05-27 19:35:21][10] Setting cache for [emails]
    [2019-05-27 19:35:40][10] Setting up schedules
    [2019-05-27 19:35:40][10] Setting up cralwer schedule
    [2019-05-27 19:35:40][10] Disabling crawler cron
    [2019-05-27 19:35:40][20] Unscheduling event wds-cron-start_service
    [2019-05-27 19:35:40][10] Found next event wds-cron-start_service at 1559078280
    [2019-05-27 19:35:40][10] Setting up checkup schedule
    [2019-05-27 19:35:40][10] Disabling checkup cron
    [2019-05-27 19:35:40][20] Unscheduling event wds-cron-start_checkup
    [2019-05-27 19:35:40][10] Setting up schedules
    [2019-05-27 19:35:40][10] Setting up cralwer schedule
    [2019-05-27 19:35:40][10] Disabling crawler cron
    [2019-05-27 19:35:40][20] Unscheduling event wds-cron-start_service
    [2019-05-27 19:35:40][10] Setting up checkup schedule
    [2019-05-27 19:35:40][10] Disabling checkup cron
    [2019-05-27 19:35:40][20] Unscheduling event wds-cron-start_checkup
    [2019-05-27 19:35:49][10] Starting a new crawl
    [2019-05-27 19:35:49][10] Sending a remote request to [https://premium.wpmudev.org/api/seo-audit/v1/start?domain=https%3A%2F%2Futterbuzz.com%2F] (start)
    [2019-05-27 19:35:50][10] Crawl started
    [2019-05-27 19:35:51][10] Setting up schedules
    [2019-05-27 19:35:51][10] Setting up cralwer schedule
    [2019-05-27 19:35:51][10] Disabling crawler cron
    [2019-05-27 19:35:51][20] Unscheduling event wds-cron-start_service
    [2019-05-27 19:35:51][10] Setting up checkup schedule
    [2019-05-27 19:35:51][10] Disabling checkup cron
    [2019-05-27 19:35:51][20] Unscheduling event wds-cron-start_checkup
    [2019-05-27 19:35:51][10] Setting up schedules
    [2019-05-27 19:35:51][10] Setting up cralwer schedule
    [2019-05-27 19:35:51][10] Disabling crawler cron
    [2019-05-27 19:35:51][20] Unscheduling event wds-cron-start_service
    [2019-05-27 19:35:51][10] Setting up checkup schedule
    [2019-05-27 19:35:51][10] Disabling checkup cron
    [2019-05-27 19:35:51][20] Unscheduling event wds-cron-start_checkup
    [2019-05-27 19:35:53][20] (Re)generating sitemap
    [2019-05-27 21:02:40][10] Forced timeout on sitemap crawl
    [2019-05-27 21:04:21][10] Setting up schedules
    [2019-05-27 21:04:21][10] Setting up cralwer schedule
    [2019-05-27 21:04:21][10] Disabling crawler cron
    [2019-05-27 21:04:21][20] Unscheduling event wds-cron-start_service
    [2019-05-27 21:04:21][10] Setting up checkup schedule
    [2019-05-27 21:04:21][10] Disabling checkup cron
    [2019-05-27 21:04:21][20] Unscheduling event wds-cron-start_checkup
    [2019-05-27 21:04:21][10] Setting up schedules
    [2019-05-27 21:04:21][10] Setting up cralwer schedule
    [2019-05-27 21:04:21][10] Disabling crawler cron
    [2019-05-27 21:04:21][20] Unscheduling event wds-cron-start_service
    [2019-05-27 21:04:21][10] Setting up checkup schedule
    [2019-05-27 21:04:21][10] Disabling checkup cron
    [2019-05-27 21:04:21][20] Unscheduling event wds-cron-start_checkup
    [2019-05-27 21:04:44][10] Setting up schedules
    [2019-05-27 21:04:44][10] Setting up cralwer schedule
    [2019-05-27 21:04:44][10] Disabling crawler cron
    [2019-05-27 21:04:44][20] Unscheduling event wds-cron-start_service
    [2019-05-27 21:04:44][10] Setting up checkup schedule
    [2019-05-27 21:04:44][10] Disabling checkup cron
    [2019-05-27 21:04:44][20] Unscheduling event wds-cron-start_checkup
    [2019-05-27 21:04:44][10] Setting up schedules
    [2019-05-27 21:04:44][10] Setting up cralwer schedule
    [2019-05-27 21:04:44][10] Disabling crawler cron
    [2019-05-27 21:04:44][20] Unscheduling event wds-cron-start_service
    [2019-05-27 21:04:44][10] Setting up checkup schedule
    [2019-05-27 21:04:44][10] Disabling checkup cron
    [2019-05-27 21:04:44][20] Unscheduling event wds-cron-start_checkup
    [2019-05-27 21:04:55][10] Setting up schedules
    [2019-05-27 21:04:55][10] Setting up cralwer schedule
    [2019-05-27 21:04:55][10] Disabling crawler cron
    [2019-05-27 21:04:55][20] Unscheduling event wds-cron-start_service
    [2019-05-27 21:04:55][10] Setting up checkup schedule
    [2019-05-27 21:04:55][10] Disabling checkup cron
    [2019-05-27 21:04:55][20] Unscheduling event wds-cron-start_checkup
    [2019-05-27 21:04:55][10] Setting up schedules
    [2019-05-27 21:04:55][10] Setting up cralwer schedule
    [2019-05-27 21:04:55][10] Disabling crawler cron
    [2019-05-27 21:04:55][20] Unscheduling event wds-cron-start_service
    [2019-05-27 21:04:55][10] Setting up checkup schedule
    [2019-05-27 21:04:55][10] Disabling checkup cron
    [2019-05-27 21:04:55][20] Unscheduling event wds-cron-start_checkup
    [2019-05-27 21:05:10][10] Starting a new crawl
    [2019-05-27 21:05:10][10] Sending a remote request to [https://premium.wpmudev.org/api/seo-audit/v1/start?domain=https%3A%2F%2Futterbuzz.com%2F] (start)
    [2019-05-27 21:05:12][10] Crawl started
    [2019-05-27 21:05:12][10] Setting up schedules
    [2019-05-27 21:05:12][10] Setting up cralwer schedule
    [2019-05-27 21:05:12][10] Disabling crawler cron
    [2019-05-27 21:05:12][20] Unscheduling event wds-cron-start_service
    [2019-05-27 21:05:12][10] Setting up checkup schedule
    [2019-05-27 21:05:12][10] Disabling checkup cron
    [2019-05-27 21:05:12][20] Unscheduling event wds-cron-start_checkup
    [2019-05-27 21:05:12][10] Setting up schedules
    [2019-05-27 21:05:12][10] Setting up cralwer schedule
    [2019-05-27 21:05:12][10] Disabling crawler cron
    [2019-05-27 21:05:12][20] Unscheduling event wds-cron-start_service
    [2019-05-27 21:05:12][10] Setting up checkup schedule
    [2019-05-27 21:05:12][10] Disabling checkup cron
    [2019-05-27 21:05:12][20] Unscheduling event wds-cron-start_checkup
    [2019-05-27 21:05:15][20] (Re)generating sitemap
    [2019-05-27 21:10:09][10] Received sitemap crawl data from remote
    [2019-05-27 21:10:31][10] Received sitemap crawl data from remote
    [2019-05-27 21:10:52][10] Received sitemap crawl data from remote
    [2019-05-27 21:11:13][10] Received sitemap crawl data from remote
    [2019-05-27 21:11:56][10] Received sitemap crawl data from remote
    [2019-05-27 21:12:16][10] Received sitemap crawl data from remote
    [2019-05-27 21:12:36][10] Received sitemap crawl data from remote
    [2019-05-27 21:12:58][10] Received sitemap crawl data from remote
    [2019-05-27 21:13:27][10] Received sitemap crawl data from remote
    [2019-05-27 21:13:44][10] Received sitemap crawl data from remote
    [2019-05-27 21:14:06][10] Received sitemap crawl data from remote
    [2019-05-27 21:14:29][10] Received sitemap crawl data from remote
    [2019-05-27 21:14:53][10] Received sitemap crawl data from remote
    [2019-05-27 21:15:15][10] Received sitemap crawl data from remote
    [2019-05-27 21:15:36][10] Received sitemap crawl data from remote
    [2019-05-27 21:15:58][10] Received sitemap crawl data from remote
    [2019-05-27 21:16:21][10] Received sitemap crawl data from remote
    [2019-05-27 21:16:42][10] Received sitemap crawl data from remote
    [2019-05-27 21:17:04][10] Received sitemap crawl data from remote
    [2019-05-27 21:17:26][10] Received sitemap crawl data from remote
    [2019-05-27 21:17:56][10] Received sitemap crawl data from remote
    [2019-05-27 21:18:22][10] Received sitemap crawl data from remote
    [2019-05-27 21:18:45][10] Received sitemap crawl data from remote
    [2019-05-27 21:19:01][10] Received sitemap crawl data from remote
    [2019-05-27 21:38:16][20] (Re)generating sitemap
    [2019-05-27 22:05:11][20] (Re)generating sitemap
    [2019-05-27 22:19:49][10] (wds-checks::focus) There are no focus keywords
    [2019-05-27 22:19:49][10] (wds-checks::title_length) Title is shorter than 50 characters
    [2019-05-27 22:19:54][10] (wds-checks::imgalts_keywords) Image alts have no keywords
    [2019-05-27 22:20:04][10] (wds-checks::focus) There are no focus keywords
    [2019-05-27 22:20:04][10] (wds-checks::title_length) Title is shorter than 50 characters
    [2019-05-27 22:20:09][10] (wds-checks::imgalts_keywords) Image alts have no keywords
    [2019-05-27 22:31:44][20] (Re)generating sitemap
    [2019-05-27 23:01:12][20] (Re)generating sitemap
    [2019-05-27 23:09:05][10] Setting up schedules
    [2019-05-27 23:09:05][10] Setting up cralwer schedule
    [2019-05-27 23:09:05][10] Disabling crawler cron
    [2019-05-27 23:09:05][20] Unscheduling event wds-cron-start_service
    [2019-05-27 23:09:05][10] Setting up checkup schedule
    [2019-05-27 23:09:05][10] Disabling checkup cron
    [2019-05-27 23:09:05][20] Unscheduling event wds-cron-start_checkup
    [2019-05-27 23:09:05][10] Setting up schedules
    [2019-05-27 23:09:05][10] Setting up cralwer schedule
    [2019-05-27 23:09:05][10] Disabling crawler cron
    [2019-05-27 23:09:05][20] Unscheduling event wds-cron-start_service
    [2019-05-27 23:09:05][10] Setting up checkup schedule
    [2019-05-27 23:09:05][10] Disabling checkup cron
    [2019-05-27 23:09:05][20] Unscheduling event wds-cron-start_checkup
    [2019-05-27 23:18:43][10] Setting up schedules
    [2019-05-27 23:18:43][10] Setting up cralwer schedule
    [2019-05-27 23:18:43][10] Disabling crawler cron
    [2019-05-27 23:18:43][20] Unscheduling event wds-cron-start_service
    [2019-05-27 23:18:43][10] Setting up checkup schedule
    [2019-05-27 23:18:43][10] Disabling checkup cron
    [2019-05-27 23:18:43][20] Unscheduling event wds-cron-start_checkup
    [2019-05-27 23:18:43][10] Setting up schedules
    [2019-05-27 23:18:43][10] Setting up cralwer schedule
    [2019-05-27 23:18:43][10] Disabling crawler cron
    [2019-05-27 23:18:43][20] Unscheduling event wds-cron-start_service
    [2019-05-27 23:18:43][10] Setting up checkup schedule
    [2019-05-27 23:18:43][10] Disabling checkup cron
    [2019-05-27 23:18:43][20] Unscheduling event wds-cron-start_checkup
    [2019-05-27 23:32:19][20] (Re)generating sitemap
    [2019-05-27 23:32:26][10] Setting up schedules
    [2019-05-27 23:32:26][10] Setting up cralwer schedule
    [2019-05-27 23:32:26][10] Disabling crawler cron
    [2019-05-27 23:32:26][20] Unscheduling event wds-cron-start_service
    [2019-05-27 23:32:26][10] Setting up checkup schedule
    [2019-05-27 23:32:26][10] Disabling checkup cron
    [2019-05-27 23:32:26][20] Unscheduling event wds-cron-start_checkup
    [2019-05-27 23:32:26][10] Setting up schedules
    [2019-05-27 23:32:26][10] Setting up cralwer schedule
    [2019-05-27 23:32:26][10] Disabling crawler cron
    [2019-05-27 23:32:26][20] Unscheduling event wds-cron-start_service
    [2019-05-27 23:32:26][10] Setting up checkup schedule
    [2019-05-27 23:32:26][10] Disabling checkup cron
    [2019-05-27 23:32:26][20] Unscheduling event wds-cron-start_checkup
    [2019-05-27 23:32:40][10] Setting up schedules
    [2019-05-27 23:32:40][10] Setting up cralwer schedule
    [2019-05-27 23:32:40][10] Disabling crawler cron
    [2019-05-27 23:32:40][20] Unscheduling event wds-cron-start_service
    [2019-05-27 23:32:40][10] Setting up checkup schedule
    [2019-05-27 23:32:40][10] Disabling checkup cron
    [2019-05-27 23:32:40][20] Unscheduling event wds-cron-start_checkup
    [2019-05-27 23:32:40][10] Setting up schedules
    [2019-05-27 23:32:40][10] Setting up cralwer schedule
    [2019-05-27 23:32:40][10] Disabling crawler cron
    [2019-05-27 23:32:40][20] Unscheduling event wds-cron-start_service
    [2019-05-27 23:32:40][10] Setting up checkup schedule
    [2019-05-27 23:32:40][10] Disabling checkup cron
    [2019-05-27 23:32:40][20] Unscheduling event wds-cron-start_checkup
    [2019-05-27 23:32:54][10] Setting up schedules
    [2019-05-27 23:32:54][10] Setting up cralwer schedule
    [2019-05-27 23:32:54][10] Disabling crawler cron
    [2019-05-27 23:32:54][20] Unscheduling event wds-cron-start_service
    [2019-05-27 23:32:54][10] Setting up checkup schedule
    [2019-05-27 23:32:54][10] Disabling checkup cron
    [2019-05-27 23:32:54][20] Unscheduling event wds-cron-start_checkup
    [2019-05-27 23:32:54][10] Setting up schedules
    [2019-05-27 23:32:54][10] Setting up cralwer schedule
    [2019-05-27 23:32:54][10] Disabling crawler cron
    [2019-05-27 23:32:54][20] Unscheduling event wds-cron-start_service
    [2019-05-27 23:32:54][10] Setting up checkup schedule
    [2019-05-27 23:32:54][10] Disabling checkup cron
    [2019-05-27 23:32:54][20] Unscheduling event wds-cron-start_checkup
    [2019-05-27 23:33:12][10] Starting a new crawl
    [2019-05-27 23:33:12][10] Sending a remote request to [https://premium.wpmudev.org/api/seo-audit/v1/start?domain=https%3A%2F%2Futterbuzz.com%2F] (start)
    [2019-05-27 23:33:13][10] Crawl started
    [2019-05-27 23:33:13][10] Setting up schedules
    [2019-05-27 23:33:13][10] Setting up cralwer schedule
    [2019-05-27 23:33:13][10] Disabling crawler cron
    [2019-05-27 23:33:13][20] Unscheduling event wds-cron-start_service
    [2019-05-27 23:33:13][10] Setting up checkup schedule
    [2019-05-27 23:33:13][10] Disabling checkup cron
    [2019-05-27 23:33:13][20] Unscheduling event wds-cron-start_checkup
    [2019-05-27 23:33:13][10] Setting up schedules
    [2019-05-27 23:33:13][10] Setting up cralwer schedule
    [2019-05-27 23:33:13][10] Disabling crawler cron
    [2019-05-27 23:33:13][20] Unscheduling event wds-cron-start_service
    [2019-05-27 23:33:13][10] Setting up checkup schedule
    [2019-05-27 23:33:13][10] Disabling checkup cron
    [2019-05-27 23:33:13][20] Unscheduling event wds-cron-start_checkup
    [2019-05-27 23:33:15][20] (Re)generating sitemap
    [2019-05-27 23:33:21][20] (Re)generating sitemap
    [2019-05-27 23:33:28][10] Setting up schedules
    [2019-05-27 23:33:28][10] Setting up cralwer schedule
    [2019-05-27 23:33:28][10] Disabling crawler cron
    [2019-05-27 23:33:28][20] Unscheduling event wds-cron-start_service
    [2019-05-27 23:33:28][10] Setting up checkup schedule
    [2019-05-27 23:33:28][10] Disabling checkup cron
    [2019-05-27 23:33:28][20] Unscheduling event wds-cron-start_checkup
    [2019-05-27 23:33:28][10] Setting up schedules
    [2019-05-27 23:33:28][10] Setting up cralwer schedule
    [2019-05-27 23:33:28][10] Disabling crawler cron
    [2019-05-27 23:33:28][20] Unscheduling event wds-cron-start_service
    [2019-05-27 23:33:28][10] Setting up checkup schedule
    [2019-05-27 23:33:28][10] Disabling checkup cron
    [2019-05-27 23:33:28][20] Unscheduling event wds-cron-start_checkup
    [2019-05-27 23:33:28][10] Setting up schedules
    [2019-05-27 23:33:28][10] Setting up cralwer schedule
    [2019-05-27 23:33:28][10] Disabling crawler cron
    [2019-05-27 23:33:28][20] Unscheduling event wds-cron-start_service
    [2019-05-27 23:33:28][10] Setting up checkup schedule
    [2019-05-27 23:33:28][10] Disabling checkup cron
    [2019-05-27 23:33:28][20] Unscheduling event wds-cron-start_checkup
    [2019-05-27 23:33:28][10] Setting up schedules
    [2019-05-27 23:33:28][10] Setting up cralwer schedule
    [2019-05-27 23:33:28][10] Disabling crawler cron
    [2019-05-27 23:33:28][20] Unscheduling event wds-cron-start_service
    [2019-05-27 23:33:28][10] Setting up checkup schedule
    [2019-05-27 23:33:28][10] Disabling checkup cron
    [2019-05-27 23:33:28][20] Unscheduling event wds-cron-start_checkup
    [2019-05-27 23:34:03][10] Received sitemap crawl data from remote
    [2019-05-27 23:34:22][10] Received sitemap crawl data from remote
    [2019-05-27 23:34:32][10] Received sitemap crawl data from remote
    [2019-05-27 23:35:11][10] Received sitemap crawl data from remote
    [2019-05-27 23:36:13][10] Received sitemap crawl data from remote
    [2019-05-27 23:36:36][10] Received sitemap crawl data from remote
    [2019-05-27 23:37:05][10] Received sitemap crawl data from remote
    [2019-05-27 23:37:28][10] Received sitemap crawl data from remote
    [2019-05-27 23:37:52][10] Received sitemap crawl data from remote
    [2019-05-27 23:38:16][10] Received sitemap crawl data from remote
    [2019-05-27 23:38:39][10] Received sitemap crawl data from remote
    [2019-05-27 23:39:00][10] Received sitemap crawl data from remote
    [2019-05-27 23:39:22][10] Received sitemap crawl data from remote
    [2019-05-27 23:39:44][10] Received sitemap crawl data from remote
    [2019-05-27 23:40:05][10] Received sitemap crawl data from remote
    [2019-05-27 23:40:25][10] Received sitemap crawl data from remote
    [2019-05-27 23:40:45][10] Received sitemap crawl data from remote
    [2019-05-27 23:41:05][10] Received sitemap crawl data from remote
    [2019-05-27 23:41:35][10] Received sitemap crawl data from remote
    [2019-05-27 23:41:50][10] Received sitemap crawl data from remote
    [2019-05-27 23:42:13][10] Received sitemap crawl data from remote
    [2019-05-27 23:42:35][10] Received sitemap crawl data from remote
    [2019-05-27 23:42:57][10] Received sitemap crawl data from remote
    [2019-05-27 23:43:18][10] Received sitemap crawl data from remote
    [2019-05-27 23:43:42][20] (Re)generating sitemap
    [2019-05-27 23:43:42][10] Received sitemap crawl data from remote
    [2019-05-27 23:44:05][10] Received sitemap crawl data from remote
    [2019-05-27 23:44:25][10] Received sitemap crawl data from remote
    [2019-05-27 23:44:47][10] Received sitemap crawl data from remote
    [2019-05-27 23:45:08][10] Received sitemap crawl data from remote
    [2019-05-27 23:45:31][10] Received sitemap crawl data from remote
    [2019-05-27 23:45:55][10] Received sitemap crawl data from remote
    [2019-05-27 23:46:19][10] Received sitemap crawl data from remote
    [2019-05-27 23:46:48][10] Received sitemap crawl data from remote
    [2019-05-27 23:47:00][10] Received sitemap crawl data from remote
    [2019-05-27 23:59:39][20] (Re)generating sitemap
    [2019-05-28 00:12:11][20] (Re)generating sitemap
    [2019-05-28 00:33:14][20] (Re)generating sitemap
    [2019-05-28 01:01:45][20] (Re)generating sitemap
    [2019-05-28 01:10:57][20] (Re)generating sitemap
    [2019-05-28 01:51:26][10] Setting up schedules
    [2019-05-28 01:51:26][10] Setting up cralwer schedule
    [2019-05-28 01:51:26][10] Disabling crawler cron
    [2019-05-28 01:51:26][20] Unscheduling event wds-cron-start_service
    [2019-05-28 01:51:26][10] Setting up checkup schedule
    [2019-05-28 01:51:26][10] Disabling checkup cron
    [2019-05-28 01:51:26][20] Unscheduling event wds-cron-start_checkup
    [2019-05-28 01:51:26][10] Setting up schedules
    [2019-05-28 01:51:26][10] Setting up cralwer schedule
    [2019-05-28 01:51:26][10] Disabling crawler cron
    [2019-05-28 01:51:26][20] Unscheduling event wds-cron-start_service
    [2019-05-28 01:51:26][10] Setting up checkup schedule
    [2019-05-28 01:51:26][10] Disabling checkup cron
    [2019-05-28 01:51:26][20] Unscheduling event wds-cron-start_checkup
    [2019-05-28 02:03:05][20] (Re)generating sitemap
    [2019-05-28 02:28:28][20] (Re)generating sitemap
    [2019-05-28 02:29:29][10] Setting up schedules
    [2019-05-28 02:29:29][10] Setting up cralwer schedule
    [2019-05-28 02:29:29][10] Disabling crawler cron
    [2019-05-28 02:29:29][20] Unscheduling event wds-cron-start_service
    [2019-05-28 02:29:29][10] Setting up checkup schedule
    [2019-05-28 02:29:29][10] Disabling checkup cron
    [2019-05-28 02:29:29][20] Unscheduling event wds-cron-start_checkup
    [2019-05-28 02:29:29][10] Setting up schedules
    [2019-05-28 02:29:29][10] Setting up cralwer schedule
    [2019-05-28 02:29:29][10] Disabling crawler cron
    [2019-05-28 02:29:29][20] Unscheduling event wds-cron-start_service
    [2019-05-28 02:29:29][10] Setting up checkup schedule
    [2019-05-28 02:29:29][10] Disabling checkup cron
    [2019-05-28 02:29:29][20] Unscheduling event wds-cron-start_checkup
    [2019-05-28 02:29:37][20] (Re)generating sitemap
    [2019-05-28 02:30:05][10] Setting up schedules
    [2019-05-28 02:30:05][10] Setting up cralwer schedule
    [2019-05-28 02:30:05][10] Disabling crawler cron
    [2019-05-28 02:30:05][20] Unscheduling event wds-cron-start_service
    [2019-05-28 02:30:05][10] Setting up checkup schedule
    [2019-05-28 02:30:05][10] Disabling checkup cron
    [2019-05-28 02:30:05][20] Unscheduling event wds-cron-start_checkup
    [2019-05-28 02:30:05][10] Setting up schedules
    [2019-05-28 02:30:05][10] Setting up cralwer schedule
    [2019-05-28 02:30:05][10] Disabling crawler cron
    [2019-05-28 02:30:05][20] Unscheduling event wds-cron-start_service
    [2019-05-28 02:30:05][10] Setting up checkup schedule
    [2019-05-28 02:30:05][10] Disabling checkup cron
    [2019-05-28 02:30:05][20] Unscheduling event wds-cron-start_checkup
    [2019-05-28 02:32:30][10] (wds-checks::focus) There are no focus keywords
    [2019-05-28 02:32:30][10] (wds-checks::title_length) Title is longer than 70 characters
    [2019-05-28 02:32:34][10] (wds-checks::content_length) Content is shorter than 300 words
    [2019-05-28 02:32:39][10] (wds-checks::focus) There are no focus keywords
    [2019-05-28 02:32:39][10] (wds-checks::title_length) Title is longer than 70 characters
    [2019-05-28 02:32:43][10] (wds-checks::content_length) Content is shorter than 300 words
    [2019-05-28 02:32:48][10] (wds-checks::focus) There are no focus keywords
    [2019-05-28 02:32:48][10] (wds-checks::title_length) Title is longer than 70 characters
    [2019-05-28 02:32:53][10] (wds-checks::content_length) Content is shorter than 300 words
    [2019-05-28 02:32:58][10] (wds-checks::focus) There are no focus keywords
    [2019-05-28 02:32:58][10] (wds-checks::title_length) Title is longer than 70 characters
    [2019-05-28 02:33:07][10] (wds-checks::focus) There are no focus keywords
    [2019-05-28 02:33:07][10] (wds-checks::title_length) Title is longer than 70 characters
    [2019-05-28 02:33:16][10] (wds-checks::focus) There are no focus keywords
    [2019-05-28 02:33:16][10] (wds-checks::title_length) Title is longer than 70 characters
    [2019-05-28 02:33:20][10] (wds-checks::content_length) Content is shorter than 300 words
    [2019-05-28 02:33:25][10] (wds-checks::focus) There are no focus keywords
    [2019-05-28 02:33:25][10] (wds-checks::title_length) Title is longer than 70 characters
    [2019-05-28 02:34:01][10] (wds-checks::focus) There are no focus keywords
    [2019-05-28 02:34:01][10] (wds-checks::title_length) Title is longer than 70 characters
    [2019-05-28 02:34:11][10] (wds-checks::focus) There are no focus keywords
    [2019-05-28 02:34:11][10] (wds-checks::title_length) Title is longer than 70 characters
    [2019-05-28 02:34:15][10] (wds-checks::content_length) Content is shorter than 300 words
    [2019-05-28 02:34:20][10] (wds-checks::focus) There are no focus keywords
    [2019-05-28 02:34:20][10] (wds-checks::title_length) Title is longer than 70 characters
    [2019-05-28 02:34:24][10] (wds-checks::content_length) Content is shorter than 300 words
    [2019-05-28 02:34:29][10] (wds-checks::focus) There are no focus keywords
    [2019-05-28 02:34:29][10] (wds-checks::title_length) Title is longer than 70 characters
    [2019-05-28 02:34:34][10] (wds-checks::content_length) Content is shorter than 300 words
    [2019-05-28 02:34:38][10] (wds-checks::focus) There are no focus keywords
    [2019-05-28 02:34:38][10] (wds-checks::title_length) Title is longer than 70 characters
    [2019-05-28 02:34:43][10] (wds-checks::content_length) Content is shorter than 300 words
    [2019-05-28 02:34:48][10] (wds-checks::focus) There are no focus keywords
    [2019-05-28 02:34:48][10] (wds-checks::title_length) Title is longer than 70 characters
    [2019-05-28 02:34:52][10] (wds-checks::content_length) Content is shorter than 300 words
    [2019-05-28 02:34:57][10] (wds-checks::focus) There are no focus keywords
    [2019-05-28 02:35:01][10] (wds-checks::content_length) Content is shorter than 300 words
    [2019-05-28 02:35:06][10] (wds-checks::focus) There are no focus keywords
    [2019-05-28 02:35:06][10] (wds-checks::title_length) Title is longer than 70 characters
    [2019-05-28 02:35:11][10] (wds-checks::content_length) Content is shorter than 300 words
    [2019-05-28 02:35:16][10] (wds-checks::focus) There are no focus keywords
    [2019-05-28 02:35:16][10] (wds-checks::title_length) Title is longer than 70 characters
    [2019-05-28 02:35:20][10] (wds-checks::imgalts_keywords) No images found
    [2019-05-28 02:35:32][10] (wds-checks::focus) There are no focus keywords
    [2019-05-28 02:35:33][10] (wds-checks::title_length) Title is longer than 70 characters
    [2019-05-28 02:35:37][10] (wds-checks::content_length) Content is shorter than 300 words
    [2019-05-28 02:35:42][10] (wds-checks::focus) There are no focus keywords
    [2019-05-28 02:35:46][10] (wds-checks::imgalts_keywords) Image alts have no keywords
    [2019-05-28 02:35:51][10] (wds-checks::focus) There are no focus keywords
    [2019-05-28 02:35:51][10] (wds-checks::title_length) Title is longer than 70 characters
    [2019-05-28 02:35:55][10] (wds-checks::imgalts_keywords) No images found
    [2019-05-28 02:35:55][10] (wds-checks::content_length) Content is shorter than 300 words
    [2019-05-28 02:36:00][10] Setting up schedules
    [2019-05-28 02:36:00][10] Setting up cralwer schedule
    [2019-05-28 02:36:00][10] Disabling crawler cron
    [2019-05-28 02:36:00][20] Unscheduling event wds-cron-start_service
    [2019-05-28 02:36:00][10] Setting up checkup schedule
    [2019-05-28 02:36:00][10] Disabling checkup cron
    [2019-05-28 02:36:00][20] Unscheduling event wds-cron-start_checkup
    [2019-05-28 02:59:25][20] (Re)generating sitemap
    [2019-05-28 03:25:50][20] (Re)generating sitemap
    [2019-05-28 04:48:06][20] (Re)generating sitemap
    [2019-05-28 05:51:35][20] (Re)generating sitemap
    [2019-05-28 06:22:52][20] (Re)generating sitemap
    [2019-05-28 08:04:24][20] (Re)generating sitemap
    [2019-05-28 08:35:44][20] (Re)generating sitemap
    [2019-05-28 09:41:41][20] (Re)generating sitemap
    [2019-05-28 10:06:47][20] (Re)generating sitemap
    [2019-05-28 10:12:00][20] (Re)generating sitemap
    [2019-05-28 11:12:48][20] (Re)generating sitemap
    [2019-05-28 11:40:42][20] (Re)generating sitemap
    [2019-05-28 12:11:57][20] (Re)generating sitemap
    [2019-05-28 12:46:21][20] (Re)generating sitemap
  • Kasia Swiderska
    • Support nomad

    Hello splaquet ,

    I'm sorry to hear that SMartCrawl is not working correctly on your site.

    I have requested our developer to take a look on your site - from what I checked there settings, memory limits are looking good so there should not be a problem to generate the sitemap. I'm waiting for the response at the moment, will let you know about what they said.

    - It still shows 1,000 URLs found, when I have the wp-config override in for 10,000

    I have these additional lines in my wp-config:
    define( 'SMARTCRAWL_SITEMAP_POST_LIMIT', 10000 );

    That is correct, Crawler will stop at 1000 links (this is limit of the service). That define limits crawler but doesn't increase the limit. So it will limit the number of URLs under 1000.

    kind regards,
    Kasia

  • splaquet
    • The Incredible Code Injector

    Kasia Swiderska , can you advise on how to best handle my sitemap then (when it works)? I have over 100,000 posts, so I'm not sure if I'm not understanding correctly or if I should seek an alternate route.

    The only thing I know is that I had an issue with SmartCrawl at one point. I was *way* too involved with 1 project, and didn't pay attention to this site for a while... until, I noticed that my Google Ads deposit went from $1200/mo to $200/mo. I was *really* relying on that income, and now I cannot figure out how to get back on top of this issue :slight_frown:

  • Kasia Swiderska
    • Support nomad

    Hello splaquet ,

    Kasia Swiderska , can you advise on how to best handle my sitemap then (when it works)? I have over 100,000 posts, so I'm not sure if I'm not understanding correctly or if I should seek an alternate route.

    I'm sorry for the delay and providing incorrect information. I have again consulted developer about this define to double check and seem that I misunderstood the original information - that define should increase the number of links in the sitemap.
    So it means it's not working on your site and I would like to have devs checking your site. Can you send us your details using our contact form https://premium.wpmudev.org/contact/#i-have-a-different-question and the template below:

    Subject: "Attn: Kasia Swiderska"
    - Site login URL
    - WordPress admin username
    - WordPress admin password
    - FTP credentials (host/username/password) OR cPanel credentials (host/username/password)
    - Folder path to site in question
    - Link back to this thread for reference
    - Any other relevant urls/info

    IMPORTANT: Please make sure you select "I have a different question" for your topic, so it doesn't go back to the forums - this and the subject line ensure that it gets assigned to me.

    Please confirm here that you send those credentials

    kind regards,
    Kasia

  • splaquet
    • The Incredible Code Injector

    Kasia Swiderska ... my apologies on the delay. was totally distracted by a few projects. and, as i watched my traffic dwindle from 6k/daily hits to about 600/daily hits... i finally had to jump back into this.

    it's painful going from $1200/mo in ad revenue to barely $100/mo :slight_frown:

    ...worth noting, I found something interesting. It was driving me nuts that the Crawler would make it all the way to 99%, but then fail. i played with the Includes, turning them on/off.

    so, when Posts, Categories & Tags are turned off (and only Pages is on), the sitemap scans and generates 100% okay! but, if any of the others are activated/deactivated, the sitemap is removed from the install and returns a 404 error.

    i'm just not able to figure out what's going on! are there simply too many posts & tags? most of my traffic arrives via Tags, so i *really* need to get this back on. $1k/mo in losses is really starting to kill me :slight_frown:

  • Panos
    • SLS

    Hi splaquet !

    I'm trying to figure if this is related to the number of posts however the testing requires adding several checkpoints in different files and I noticed that site has some visits so I wouldn't like to risk having your site down. Is it possible to create a staging site which is the exact copy of your site so I could do the tests there? You can ask your host if they provide some automated method to create a staging site.

    If this is possible, could you send in admin and ftp credentials? You can send those in privately through our contact form: https://premium.wpmudev.org/contact/#i-have-a-different-question

    Send in:Subject: "Attn: Panos Lyrakis"

    - Admin login:
    Admin username
    Admin password
    Login url

    - FTP credentials
    host
    username
    password
    (and port if required)

    - link back to this thread for reference

    Kind regards!

  • splaquet
    • The Incredible Code Injector

    Panos ... can you grab the credentials from Kasia Swiderska ? I'd honestly have to create a new set of them, because I didn't save the passwords.

    ###

    but, as a major discovery... i found that the sitemap *started identifying pages* when i referenced the subfolder install in Google Search Console. (still not identifying Tags in the sitemap though)

    I found that super odd though. I can see the sitemap.xml just fine at https://utterbuzz.com/sitemap.xml ...but Google kept reporting 0 pages

    when i used https://utterbuzz.com/ubms/sitemap.xml ... it picked up 15k+!!! wowie... so odd.

Thank NAME, for their help.

Let NAME know exactly why they deserved these points.

Gift a custom amount of points.