[SmartCrawl] URL's not added to SiteMap

SmartCrawl on my website shows that there are 329 URLs missing in the site map. I tried running a plugin conflict test and cleared Hummingbird cache, but still the last crawl date is of August 18, 2018 10:59 pm.

I tried adding the URL's using the SmartCrawl panel and then cleared the cache and ran a new crawl. I did this several times but still the issue persists.

I would like this to get sorted.

Thanks

  • Adam Czajczyk

    Hi Andi Schwartz

    I hope you're well today and thank you for your question!

    I checked your site and the missing URL issues. Those URLs that are reported as missing from sitemap are already in the sitemap: once the "Add to sitempa" (for one URL or the "all" button at the bottom) is clicked they are simply appended to the existing sitemap. That doesn't require another crawl or seo checkup.

    On a "Dashboard -> Home" page in the "Sitemaps -> SmartCrawl" meta box you can see that the map has been re-generated today and that it contains currently 1443 items. Additionally, you can also open the sitemap.xml itself in the browser and double-check if these URLs are there. I did check couple of them (randomly selected) to confirm that they were added.

    However, this will not change what the "SmartCrawl -> Sitemap" section says because, well, that actually does require crawl - even despite the URLs are already added. So, the question is why that crawl doesn't start and I think around the time the last crawl was completed (in August) or later the site's been switched to SSL (from http:// to https://), is that right?

    When trying to run the crawl I noticed mixed content errors in the browser console that caused some important internal calls to be blocked. I checked site settings and it's not configured to run over HTTPS connection even though it is using SSL. That said, please go to the "Settings -> General" page in your site's back-end and make a small change there:

    make sure that both "WordPress Address (URL)" and "Site Address (URL)" options are changed to use "https://" in URL instead of current "http://".

    This is required for a valid and proper SSL configuration of WordPress. Once that's done, clear all the caches on site and server and try running "New Crawl" from "SmartCrawl -> Sitemap" and see if it works. If not, wait a couple of hours (our crawlers might need some time to "update themselves") and check again. If it still doesn't work, let me know and I'll investigate it further.

    Best regards,
    Adam

  • Predrag Dubajic

    Hi Andi,

    We were looking into the SmarCrawl issue and the scan is working when started from our end but it isn't updating data on plugin itself on some installation.
    We have been able to replicate this on one of our installations and our developers are currently looking into it further.

    As for the Smush CDN issue, this is still in beta and our devs are currently working on 3.1 version that will come with further improvements for this.
    It also seems that your CDN bandwidth limit has been reached and we're also working on notifications about being close to the limit and options to increase available bandwidth for the account.
    One of the improvements that are being worked on is making sure that server images are being served, instead of CDN ones, once bandwidth has been used.

    Best regards,
    Predrag

Thank NAME, for their help.

Let NAME know exactly why they deserved these points.

Gift a custom amount of points.