New to the community? Start here

Update of Sitemap and Forcing Search Engine Notification

Hi,

Update of Sitemap and Forcing Search Engine Notification on the dashboard of the subsite https://shop.roadmapstores.com can be started but does not complete anymore. Even after more than one hour staying on the page and not refreshing the dashboard page there was no progress visible. I have granted remote access to admin area for one week. Please let me know what causes the issue. The last update was on 8th of January 2018.

Thanks

Rod

Mon, 22 Jan 2018 19:33:08
More SmartCrawl Pro discussions
SmartCrawl Pro
  • James Morris
    • WordPress Enthusiast

    Hello Rod,

    I hope you are well today.

    I took a look at your site and what I’m seeing indicates you have a conflict on the shop subsite.

    When I checked your primary site, I see SmartCrawl is updating correctly and notifying search engines. The last update timestamp was

    It was last updated on January 22, 2018, at 1:15 pm.

    However, on your shop subsite, it shows the last update was Jan. 8th and I observed the symptoms you described. Checking the JS Debug Console in Chrome DevTools gave a little insight.

    Failed to load resource: the server responded with a status of 504 ()

    I checked the active plugins on both sites and I see the shop subsite has 15 more plugins activated on it than your primary site. This leads me to believe you have a plugin conflict.

    In order to test this, I need you to perform the following steps, please:

    – Go to your Shop Subsite WP Admin -> Plugins

    – Deactivate all plugins that are only active on that subsite

    – In a new tab, go to Subsite WP Admin and press CTRL+Shift+i on your keyboard to bring up the Google Chrome DevTools window. Click on the “Console” tab to bring up the JS Debug Console.

    – Click on the “Update sitemap now” link in the SmartCrawl Sitemap Dashboard Widget.

    – If no error displays in the JS console, enable one of the previously disabled plugins, refresh the subsite Dashboard and try the “Update sitemap now” link again.

    – Repeat until the error is produced.

    This will help you to narrow down which plugin is conflicting with SmartCrawl.

    Let us know if you have any further questions. We’ll be happy to help! :slight_smile:

    Best regards,

    James Morris

  • Rod
    • Site Builder, Child of Zeus

    Hi James,

    thanks for the fast reply. When it comes to an operation like ‘Deactivating ALL Plugins’ it always means a lot of ‘Re-Setting’ after everything is done. Not to mention the time for ‘Site Is Down For Maintenance’. It is always a horrible task. On a live site like my shop with such a large amount of mostly essential plugins and lots of settings, I don’t feel comfortable to start with ALL plugins.

    Instead, I thought about to compare the list of the other two sites which work and then grab the plugins list left taking a closer look at the most suspicious ones. It should lead to the same result. What do you think?

    Thanks

    Rod

  • James Morris
    • WordPress Enthusiast

    Hello Rod,

    Perfectly understandable, and yes, eliminating the common plugins would be the best course of action. We know it works on your primary domain. So, it’s just identifying what is different on this one subsite that’s causing the problem.

    I look forward to the results of your test.

    Best regards,

    James Morris

  • Rod
    • Site Builder, Child of Zeus

    Hi James,

    I didn’t start yet to investigate the plugins because there are some more timeout issues I am monitoring for a while now only on the shop subsite. With ever-growing amounts of products i.e. editing the main menu of the shop site became almost impossible to access resulting in 504 timeout error. Even if the menu doesn’t have an excessive amount of items.

    And there was another amazing fact regarding the case we are talking about. Yesterday the server admin restarted the whole instance and right after it was up and running again date and time of updating sitemap and search engine notification on the related dashboard widget were set to the one when the machine was back. BUT trying to update manually thereafter showed the same issue again we are discussing.

    As there is a major update available of the underlying Linux OS now as well as a related update of the panel I have decided to wait for the results when this is done. We will start setting up a new instance and then migrate the whole site to it within the next days.

    I’ll let you know the results. Remote access is still available so if you want to take a short look at the recently updated widget of the SmartCrawl plugin, just go ahead.

    Thanks, I’ll keep you updated.

    Rod

  • Rod
    • Site Builder, Child of Zeus

    Hi James,

    I have detected an error message inside my Google search console saying that there is a mistake with the sitemap. I have attached a screenshot and must explain the terms as Google is keeping the German user interface my browser. Sorry about that.

    There is a value saying 50.000 pages (Webseiten) exactly submitted (Eingereicht). The blue arrow points there. But only 8.386 got indexed so far (red arrow).

    But below it is showing one mistake (1 Fehler). The description (Beschreibung) says that the sitemap contains TOO MANY (zuviele) URLs. As a solution it is then recommended to generate MULTIPLE sitemaps each containing up to 50K URLs and resubmit all.

    Is there a way to configure this with Smart Crawl? Might this be the root cause of the issue we discuss here?

    Thanks

    Rod

  • James Morris
    • WordPress Enthusiast

    Hello Rod,

    Taking a look at your sitemap setup, it looks as though you are including an excessive amount of taxonomies. There are a few there you definitely would not want to include in your sitemap.

    The taxonomies marked in red should not be included in your sitemap. The taxonomies marked in yellow *may* be useful in some situations, but should be included only with caution. .

    This would be why you are seeing so many URLs in your sitemaps. It also will contribute quite a bit to the timeout issues if you have SmartCrawl set to automatically update the sitemap with each post/page/product update. You can alleviated this by disabling unneeded taxonomies and by setting sitemap updating to manual only.

    I hope this clarifies a bit.

    Best regards,

    James Morris

  • Rod
    • Site Builder, Child of Zeus

    Hi James,

    I have disabled the taxonomies both marked red and yellow and set updating sitemaps to manually. But I guess this is just a small improvement to resolve the issue. I have attached another screenshot of the recent sitemap on the shop subsite. This was updated on 27th of January after restarting the server. But it still isn't possible to update within the dashboard widget. As I am migrating to a new instance recently I'll take care of that later.

    As you can see the sitemap now contains 116364 items. WooCommerce is counting 69890 products to date. The higher number of items is probably including the products' variations (also counting as products) and some of those items like obsolete taxonomies resulting from the former settings. Each product has its own URL. This number will increase ongoing up to some 3 Mio. products in the next months.

    As a result, it should certainly be possible to generate multiple sitemaps each not containing more than 50K items. The question now is, if SmartCrawl can do that? If not I guess I might have to find a different solution for the shop. What do you think?

    Thanks

    Rod

  • James Morris
    • WordPress Enthusiast

    Hello Rod,

    I’m terribly sorry for the delay in getting back to you.

    Unfortunately, it is not currently possible to break sitemaps down into multiple sitemaps. However, I’ve spoken with the Lead Dev on this project and they feel this would be a good addition. I’m tasking this up as a feature request to be added to future versions.

    Best regards,

    James Morris

  • Rod
    • Site Builder, Child of Zeus

    Hi James,

    no problem, thanks for the update. I also think this would be a good idea to break sitemaps into multiple sitemaps. On the related shop subsite, meanwhile, I have set up a single SEO premium plugin which offers this feature. I’ll keep this solution until Smart Crawl comes up with it.

    So we can mark this case as resolved.

    Thanks

    Rod

  • Rod
    • Site Builder, Child of Zeus

    Hi James,

    I’ve reopened this topic because as a result of the missing feature discussed (sizeable sitemaps in Smart Crawl) I have installed a third party premium SEO plugin which is offering to define the size of the sitemap. But this plugin does not support multisite so I installed and activated it just on the related shop subsite in my network.

    But now I am thinking about the overall result regarding generation of sitemaps. As Smart Crawl is network activated it still shows the related widget at the shop subsite’s dashboard. Does this mean there might exist kind of a related conflict with the sitemap generated by the new plugin? The URL of the new sitemap.xml is located in the shop subsite’s home directory.

    Please let me know your point of view on this issue.

    Thanks

    Rod

  • Ash
    • WordPress Hacker

    Hello Rod

    Using two sitemap plugins can create conflict. But as you have activated that plugin in a specific subsite, now what happens if you visit that subsite specific sitemap, then which sitemap do you see? If you see sitemap from that plugin as well as sitemap works for other subsites, then I think you should be good to go :slight_smile:

    Have a nice day!

    Cheers,

    Ash

  • Rod
    • Site Builder, Child of Zeus

    Hi Ash.

    I must admit that it is difficult for me at the moment to follow up completely. There is one sitemap.xml in the ‘uploads’ folder of the main domain. This one was generated by Smart Crawl in the beginning. It was already not very clear to me why it was generated there but I had a support ticket going on that time and we just set that path in the admin area of the Smart Crawl plugin. After that the widgets in all subsites got displayed correctly each showing the subsite’s items. But then this ‘limit’ problem with the 50K items showed up in the shop subsite and could not be resolved. In the Google search console appeared a message that sitemap files should be split into 50K items max. That is why I have installed this other plugin at the shop subsite only. But I cannot say where this sitemap.xml file is located. I guess I simply don’t understand the overall multisite file structure. How can I have a specific look at the shop subsite’s home directory, where the new sitemap.xml file is supposed to appear?

    Thanks

    Rod

Thank NAME, for their help.

1 Point
5 Points
15 Points
Other

Let NAME know exactly why they deserved these points.

Gift a custom amount of points.