Google Search Console Errors

On Google Search Console it says:

When we tested a sample of the URLs from your Sitemap, we found that some URLs were not accessible to Googlebot due to an HTTP status error. All accessible URLs will still be submitted.

HTTP Error: 404 URL: /wisdom-038/ - Nov 11, 2016
HTTP Error: 404 URL: /wisdom-011/ - Nov 6, 2016
HTTP Error: 404 URL: /faq/10-why-no-free-trial/ -Nov 2, 2016

I want all wisdom posts and faq’s to be available. Can you tell me what is stopping googlebot from accessing them. How do I fix these errors?

Support Access for christiangays.com is available for four more days.
Thank you.

  • James Morris
    • WordPress Enthusiast

    Hello Mary,

    I hope you are well today.

    I just checked the 3 URLs you shared. The first two could not be found (404) when I browsed your site either logged in or not. So, those pages do not exist on your site as far as I can tell. If they do, you may want to check your permalinks and ensure they are correct.

    The 3rd URL does exist and is browsable either logged in or not. So, this is possibly just a false 404 on Google's part. Looking at the time-stamps on those errors, I see they are from a couple months ago and could have been during a time you were doing heavy development and the pages were not available temporarily.

    In Search Console, if you have confirmed that a link is valid, you can mark it as fixed. You can also "Fetch as Google" to make sure your content is properly rendered. See screenshot: https://goo.gl/Zt1F6c

    If your sitemap includes links to content that no longer exist, you may need to manually regenerate your sitemap. How this is done depends on which tool you're using to create your sitemaps.

    If you have a large number of 404 errors due to moving around your site structure a bit, you may want to consider a redirect plugin to redirect users back to your homepage, or a page of your choosing, to improve the user experience a bit. Something like the following might work nicely: https://wordpress.org/plugins/all-404-redirect-to-homepage/

    I hope this clarifies a bit.

    Happy New Year! :slight_smile:

    James Morris

  • Mary
    • Wisp of the WP Loop

    Thank you James. I rebuilt my site over a year ago but for some reason Google is still searching for content from the old website which no longer exists. I am trying to use robots.txt to stop it from trying to index. Can you tell me please if this is correct. i.e. Will
    Disallow: /store/
    Disallow: /stories/
    stop all files that are in the "store" and "stories" folders?

    This is the robots.txt file I'm using. Please let me know if there are problems with it.

    User-agent: Mediapartners-Google

    Sitemap: https://christiangays.com/sitemap_index.xml

    Disallow:

    User-agent: *

    Disallow: /?cat=
    Disallow: /?p=
    Disallow: /?page_id=
    Disallow: /?topic=
    Disallow: /2-day-test/
    Disallow: /s2-login-welcome-page/
    Disallow: /aboutus/
    Disallow: /activate/
    Disallow: /advanced-search/
    Disallow: /articles/
    Disallow: /author/
    Disallow: /blog/
    Disallow: /category//
    Disallow: /cgi-bin/
    Disallow: /chat/
    Disallow: /chat_hottopics.shtml
    Disallow: /choose-membership-level/
    Disallow: /cms/
    Disallow: /cometchat/
    Disallow: /complete-your-profile
    Disallow: /donation-confirmation/
    Disallow: /donations/
    Disallow: /dr-author/
    Disallow: /ebooks.shtml
    Disallow: /email2.php/
    Disallow: /e-newsletter-unsubscribe-successful/
    Disallow: /error_log/
    Disallow: /forgotpass.php/
    Disallow: /forums/
    Disallow: /ftp.
    Disallow: /humour/
    Disallow: /invboard/
    Disallow: /links/
    Disallow: /marriage/
    Disallow: /members/
    Disallow: /membership-account/
    Disallow: /membership-join/
    Disallow: /membership-renewal-upgrade/
    Disallow: /motion/
    Disallow: /my-forum-subscriptions/
    Disallow: /newsletters/
    Disallow: /osdate/
    Disallow: /other/
    Disallow: /personals
    Disallow: /prayers.shtml
    Disallow: /products/
    Disallow: /protected-content/
    Disallow: /registration-complete/
    Disallow: /reply/
    Disallow: /scrolling-gallery/
    Disallow: /signin/
    Disallow: /smf/
    Disallow: /store/
    Disallow: /stories/
    Disallow: /test-registration/
    Disallow: /tidbits.shtml
    Disallow: /tmp.
    Disallow: /topic-tag/
    Disallow: /trans.shtml
    Disallow: /using-joomla/
    Disallow: /wp-login.php
    Disallow: /wp-signup.php

  • James Morris
    • WordPress Enthusiast

    Hello Mary,

    I hope you are well today.

    I see a couple problems with your robots.txt which may be causing your problem.

    First, the the beginning of your file is wrong.

    This:

    User-agent: Mediapartners-Google
    
    Sitemap: https://christiangays.com/sitemap_index.xml
    
    Disallow:

    * User-agent: Mediapartners-Google is only for Google AdSense. See: https://support.google.com/webmasters/answer/1061943?hl=en
    * Your Sitemap directive should fall under the User-agent: * directive so all search engines can find it.
    * Your query string Disallow directives are not setup right. See: https://sanzon.wordpress.com/2008/04/29/advanced-usage-of-robotstxt-w-querystrings/

    Your directory Disallow directories look OK, as far as I can tell, but you might want to run a live test of whether URLs are allowed or not using the following tool: https://technicalseo.com/seo-tools/robots-txt/

    You can learn more about robots.txt in this Google Tech doc: https://support.google.com/webmasters/answer/6062608?hl=en&visit_id=1-636199310609154082-750294761&rd=1

    Google Search Console also has it's own robots.txt tester that may be useful: https://support.google.com/webmasters/answer/6062598?hl=en

    I hope this clarifies a bit. :slight_smile:

    Best regards,

    James Morris

  • Mary
    • Wisp of the WP Loop

    Thank you James. I am more confused than ever.

    Based on what I read I just added * to
    Disallow: /?cat=*
    Disallow: /?p=*
    Disallow: /?page_id=*
    Disallow: /?topic=*

    When I did the tests it said "allowed" so I think I'm OK unless you have further input.

Thank NAME, for their help.

Let NAME know exactly why they deserved these points.

Gift a custom amount of points.