Fix 404 not found urls automatically

I had I have switched to yoast from squirrly SEO on one of my sites (as it can take over the settings) before going on to smartcrawl (as it can import from yoast). However, now that I'm checking the crawl errors - google complains about the oddest items as access denied.
and I have a bunch of not founds from plugins I disabled.
The not founds are cache items and access denieds are files that work behind the scenes inside plugins - how do I get rid of these? (I'm happy to switch to smarcrawl now if it will help, I don't know what will make things more / less complicated?)

If Smartcrawl does not have any automated way to deal with these errors please add this as a feature request.

  • Adam Czajczyk

    Hello Symi,

    I hope you're well today and thank you for your question!

    Fixing any "404" URL automatically is not that easy as it might seem. It wouldn't be a problem to set automatically a "general"/"overall" redirect for such URL's (which is actually a solution, apart from not including them in a sitemap) but the there's no way of knowing - automatically detecting - if those 404 responses from these URLs are permanent or temporary (for example if a page went down due to some other error) or if they are intentional or not.

    In other words: a 404 status always mean "not found"/"not existing" but doesn't give any reason/explanation why and that in most cases makes it impossible to automatically distinguish between "legitimate" 404 (like a real "404 Not Found" core WP page) and those "broken links".

    Also, the fact that such links are reported would mean that they were indexed by search crawlers. They would actually got "de-indexed" over time automatically but the way to deal with them is to make sure that

    1) they are not linked to
    2) they are not in a sitemap
    3) they are redirected to either "core" 404 Not Found page or to equivalent/relevant existing replacement pages or e.g. to homepage (which one - that's a matter of a "strategy" you'd like to take on that).

    As for #1)

    I think in your case that shouldn't be much of an issue, judging by the links reported on the screenshot from chat; However, it would still be recommended to review content of the site carefully to makes sure that links to those "404 URLs" are not posted anywhere on site and if they are, remove them from there. Also make sure that caches on site and server (if any) are fully cleared after that.

    As for #2)

    If the above aspect is taken care of, simply re-generating a sitemap and then re-submitting it to Google (and giving it some time to re-crawl) should suffice

    As for #3)

    Finally, it's a matter of setting redirects and that can be done in SmartCrawl. You only need to enable the "Advanced Tools" module on "SmartCrawl -> Settigns" page and you can then set redirects on "SmartCrawl -> Advanced Tools -> URL Redirection" page.

    If you have any additional questions, let me know please.

    Best regards,
    Adam

Thank NAME, for their help.

Let NAME know exactly why they deserved these points.

Gift a custom amount of points.