Need Advice Re 500 Errors

I started my ChristianGays.com website in 2000. By 2015 it had over 8,000 pages. I decided to rebuild the site as a WordPress site so I deleted the old site and started from scratch.

I assumed that once the old pages were gone, that Google & Bing would no longer see them in a search. I was wrong. I keep getting 500 errors for pages not found.

Obviously I can’t go through the 8000 pages individually. Is there a way to stop Bing and Google from searching for the pages from my old site?

I have new sitemaps but other than that, I don’t know the best way of handling these errors. Would appreciate advice.

Thank you!

  • Mary
    • Wisp of the WP Loop

    I also had a major problem. My View Profile, Edit Profile, Forum Subscribe etc links were all going to home page. I have had to deactivate the plugin.

    I would like to use the plugin but need it to not block my legitimate links.

    Support access is granted.

  • Adam Czajczyk
    • Support Gorilla

    Hello Mary!

    The plugin suggested by Nastia shouldn’t affect your legitimate links/URLs unless there’s some kind of conflict. I wouldn’t worry that much about it though: Google will finally quit on trying to crawl these URL’s since it’s getting 404 response instead of proper page. It should remove those “legacy” URL’s from its indexes after a while.

    In order to speed up this process just make sure that you do have a Google Webmasters account and that you uploaded current sitemap(s) of your site there. It should then re-index you website following URLs from sitemaps.

    Best regards,

    Adam

  • Dimitris
    • Support Star

    Hey there Mary,

    hope you’re having a beautiful day and don’t mind chiming in! :slight_smile:

    Did you try to access your Search Console in Google Webmasters like Adam mentioned?

    https://www.google.com/webmasters/

    In this control panel you can re-submit your newest sitemap, check and fix crawl errors etc.

    https://support.google.com/webmasters/

    https://moz.com/blog/a-beginners-guide-to-the-google-search-console

    On the other hand, which plugin do you use to create sitemaps and are there any other SEO plugins active in your installation? Please advise! :slight_smile:

    Take care,

    Dimitris

  • Mary
    • Wisp of the WP Loop

    Thank you Dimitris. I appreciate you chiming in. Yes, I have been submitting new sitemaps regularly for the past 9 months. I go into crawl errors, set the page to view 1000 per page, tick the boxes that they are “fixed”, and within a few days they are back again.

  • Adam Czajczyk
    • Support Gorilla

    Hello Mary!

    Thank you for your response, this is very helpful.

    Marking those broken links as “fixed” in Google Webmaster Console will not help. This way you only let Google know that you actually fixed the site and the “broken” link is again valid. That’s why Google keeps trying to re-index them and why these errors get back again. The first step then would be to stop marking these links as fixed.

    Then the question comes up: where does Google take these “broken” URLs from?

    1. Most of them are URLs that were already indexed and simply exist in Google database. Google attempts to visit them in order to re-index content from time to time. When this happens, Google hits “not found” page (404 error) and a message of “broken” link shows up and that links goes on the crawl errors list of Google Wembaster console.

    2. Some of these links may have already been removed from Google index but crawlers still visit your site and other sites around the web. There’s a chance that those “broken” links are still somwhere there in the content (e.g. a non-existing page is still linked to from post content on your or 3rd party site).

    That being said, I think another approach would be necessary. First thing would be to identify sources of those broken links. The easiest solution is to give a “Broken Link Checker” plugin a go. Take a look here please:

    https://wordpress.org/plugins/broken-link-checker/

    This plugin scans your site and reports all the broken links to you. Your task then is to follow that report and either remove or update reported links on your site. This alone should help a bit.

    Next step would be similar but more “time consuming” unfortunately but should let you identify any links that the plugin may have missed for some reason and would also let you find if there are any incoming “broken” links from 3rd party sites. Here’s a detailed guide on what should be done:

    http://wpsites.net/seo/how-to-fix-404-page-not-found-crawl-errors-using-google-webmaster-tools/

    Again, if you again find that any of your sites/pages contain a “broken link”, make sure that you removed it or replaced with a valid one. If you find out that external sites/pages are linking to your site(s) with broken URLs you may want to try to get in touch with admins of those sites and ask them to remove/update those links. This unfortunately rarely works but it’s always worth giving a try.

    As I explained before, please do not mark any of those crawl errors in Google Webmaster console as “fixed” during that process.

    Once this is done we could then try to “teach Google” about that those broken links that are still in their database are no longer relevant but that would be the last step and I think it would be best if you could start with what I suggested above. After that, let me know please and I’ll be more than happy to assist you further.

    Have a nice day!

    Adam

  • Mary
    • Wisp of the WP Loop

    I had already checked for broken links. I do that regularly but added the plugin as you suggested – good idea. It is install on my main site and two sub-domains (had to install individually since plugin doesn’t support multisite). I have no broken links.

    I went into Webmaster Tools again and today I only have 587 not founds so it is getting better. They are all links from the old site to pages on the old site, meaning that I can’t forward or change them to a “correct” link.

    So just to verify, you say that I should NOT click these as “fixed” then?

    I watched the video and read the second link, but it seems that there is nothing I can do but wait for Google to remove them in their own good time. Is that correct?

    I tried entering the code in child theme functions.php file to redirect to home page, but it doesn’t seem to work. This is one of the URL’s that is not found https://christiangays.com/store/product_info.php?products_id=140

    A lot of URL’s not found are coming from my old “store”. Can I redirect them all somehow to the new store?

  • Adam Czajczyk
    • Support Gorilla

    Hello Mary!

    Thank you for your response!

    It’s good to know that it’s getting better :slight_smile:

    As I explained before, I wouldn’t recommend clicking “fixed” because it’s a way to say Google crawlers that the link shouldn’t be removed because it’s again working fine. Google then re-crawls it and again puts it on “broken links list”. I suspect that’s part of the reason why you weren’t able to get rid of these links for such a long time :slight_smile:

    but it seems that there is nothing I can do but wait for Google to remove them in their own good time.

    That’s a yes and no. If left to its own devices in an “as is” state, Google should eventually stop crawling these links and just remove them from index:

    To remove content or a URL from Google search permanently you must take one or more of the following additional actions:

    Remove or update the actual content from your site (images, pages, directories) and make sure that your web server returns either a 404 (Not Found) or 410 (Gone) HTTP status code. Non-HTML files (like PDFs) should be completely removed from your server. (Learn more about HTTP status codes)

    Block access to the content, for example by requiring a password.

    Indicate that the page should not to be indexed using the noindex meta tag. This is less secure than the other methods.

    https://support.google.com/webmasters/answer/1663419?hl=en

    In case of WordPress those requirements should be met “automatically” but it may be good to check whether those broken links indeed return 404 HTTP status. In order to check it I visited your main site and added some random string to its address to make sure that I’m trying to visit a non-existent page. I was then redirected to the homepage with “301 Moved Permanently” status.

    That’s a bit different but should do the trick as well. The “301” status doesn’t tell browser (Google crawler in this case) that the site doesn’t exist but instead it says that it has been moved to another place. As a result those “broken links” wouldn’t be removed from index but still, they should “stop being broken” over time.

    Alternatively, you could switch off that redirect and then we would make sure that the “Not found” page is returning proper 404 status. It should be doing that “out of the box”.

    Best regards,

    Adam

  • Mary
    • Wisp of the WP Loop

    Sorry Adam but I’m really confused. My old site was not WordPress. It was created before WordPress existed and almost all of my pages were .shtml pages.

    When I created the new WordPress site I deleted everything from the christiangays.com domain name and put everything on the christiangays.ca domain so that I can use it for a reference. So in actuality the pages do exist but on a different domain.

    I had read the info from Google before, but couldn’t see a way to use that information in my case.

    Other than doing nothing, is there anything else that I can do?

    Is there a way to redirect all of these pages to the new site without doing each page individually?

    Thank you.

  • Adam Czajczyk
    • Support Gorilla

    Hello Mary!

    I admit you got me confused now as well. Could we get back a bit for a second?

    The site we are referring to is a ChristianGays.com. That site wasn’t a WordPress site back in the days and it used links ending with “.shtml”. One day you removed everything from there (it’s a simplification but let’s stick to it) and in this place set a new site, the one that’s there now.

    Then you started getting a lot of “not found” errors on that new site. Is that correct so far?

    That’d be an (imaginary) example:

    – there was a page on “old” site: christiangays.com/mypage.shtml

    – there’s no such page now

    – you’re getting report of “christiangays.com/mypage.shtml” not found

    Am I still on the right track?

    If so, here’s what’s happening:

    1. Google indexed your “old” site once so it knows about “christiangays.com/mypage.shtml” URL

    2. It’s trying to visit that URL but your “new” (WordPress) site currently is redirecting it to the “christiangays.com” (homepage), returning HTTP Status of “301 Moved Permanently”

    3. This way it informs Google that the “christiangays.com/mypage.shtml” page that it is looking for is now the “christiangays.com” page.

    Is it a bit more clear? :slight_smile:

    The christiangays.ca site is not relevant here as for Google it’s a separate site.

    Getting back to the Google Webmaster Console: when google crawler hits the “christiangays.com/mypage.shtml” page which in reality doesn’t exist anymore, it gets aforementioned 301 status. It should overtime stop telling you that this link is broken.

    That is, unless you keep marking it as “fixed” because marking it fixed suggests: “I brought back the “christiangays.com/mypage.shtml” page as it was before, please Google check it again”. Then bam – it all starts again, in a loop.

    So, if you have not marked that “fixed”, Google will keep hitting that URL but should eventually learn that it’s not “broken”.

    That’s however just one side of the story. Because it will not fix anything on your WordPress site and as long as that “christiangays.com/mypage.shtml” URL will be visited by crawlers your new WordPress site will still note it as a “not found”. It will stop when Google stops visiting it, as I explained above.

    Does that make any sense to you?

    I realize that this is not quite “intuitive stuff” but I hope I made it at least a bit clearer :slight_smile: If you require further explanation/advise on this let me know and I’ll be happy to continue assisting you.

    Have a nice day!

    Adam

  • Mary
    • Wisp of the WP Loop

    EXCELLENT EXPLANATION!!!! You have got it EXACTLY, and I now believe that I totally understand.

    Thank you Adam. I will close this ticket now, not “fix” any more bad links on Google Webmaster and try to be patient until they stop indexing.

    I’m amazed that you totally GET IT from my explanation. Bravo!

    Thank you again!

Thank NAME, for their help.

Let NAME know exactly why they deserved these points.

Gift a custom amount of points.