Robots.txt

I can't find the robots.txt file. It's causing the problem below and needs to be removed. Any help is appreciated.

erealestate.com is currently blocked by a robots.txt file that has been uploaded to the site: http://erealestate.com/robots.txt. Should the file be stricken or adapted to allow the crawler ia_archiver, crawling for the Wayback Machine can continue. Until then, the presence of the site is considered an express request to exclude the specified domain from the Wayback Machine.

  • drmike
    • DEV MAN’s Mascot

    Your robots.txt file is fine. WordPress now generates it on the fly.

    Actually it looks like you have the blog in question set to not be indexed by search engines as this is appearing within the blog's outputted source.

    <meta name='robots' content='noindex,nofollow' />

    Take a look at the Privacy settings and make sure you;ve set it to be indexed by search engines. If it is, the line above maybe hardcoded into the theme's header.php file and you;ll have to remove it.

    I have a feeling that it's hardcoded as if you had set your blog to not be indexed, the robots.txt file would also reflect that and it's not.

    Hope this helps,
    -drmike

    edit: Oh, and I;ve moved this thread to the Discussion subforum. As per the forum descriptions on the main page, Andrew would rather have the Plugins Support subforum only for the plugins available here.

  • drmike
    • DEV MAN’s Mascot

    Um, no disrespect to him but James' example is rather lacking and incorrect. It has a couple of dupes and I for one would rather include the mu-plugins subdirectory as well.

    Heck, if you;re going to do that, I'd just place a robots.txt file within root and put in there what you want. I do agree with James though that 99% of your endusers aren;t going to understand what a robots.txt file is anyway unless you;ve got a specific niche that you cater towards.

    The kbrobots plugin he links to is another option and what we use on our installs. Do a search of the forums here though for kbrobots as I had a security concern about it that really never got answered. (I;ve got 2 minutes of net time left.)

  • Barry
    • DEV MAN’s Mascot

    Can you check with an FTP program and see if you actually have a robots.txt file in the root of your WP site as an actual file existing will override the WP created version.

    The privacy settings you mention are the ones in Settings -> Privacy, correct?

  • Barry
    • DEV MAN’s Mascot

    Yes, a robots.txt in the root would affect all sites.

    Sorry, just looking at the code in WP - that robots.txt seems to actually be correct as far as WP is concerned.

    If it looked like this:
    User-agent: *
    Disallow: /

    Then it would block access.

    Can someone more up on robots.txt confirm this (or not) the WP function is in wp-includes/functions.php - line 1779 function do_robots()

  • IntegraLLC
    • Flash Drive

    Hi there- it looks like I'm having the same problem at http://gbiportal.net . I did have my privacy settings set to hide from search engines, but I've switched it to be visible to everyone. However, I'm still getting "blocked by robots.txt" message from Google. I have tried creating a new robot.txt file, deleting any that I see, but still the same thing.
    I am now not seeing any robot.txt files (from my ftp) but can still see User-agent: *
    Disallow:
    at http://gbiportal.net/robots.txt

    What's going on?
    Where should I look for the : <meta name='robots' content='noindex,nofollow' /> mentioned in DrMike's first response?

    I'm using BP-Daily theme, in case it is relevant.

    Thanks,
    Laurie

  • IntegraLLC
    • Flash Drive

    So, I never figured out what was wrong. I ended up finding a plugin that wrote a new robots.txt file for me (although why the one I was trying wasn't working is beyond me). That still didn't work. BUT, I then ran the site through ismyblogworking.com - this verified that the robot.txt file was fine. Soon after that, I checked with google again, and this time it found it worked fine! Not sure what was the winner - new plugin's robot file, waiting several hours since I reset the privacy settings (somewhere I saw it could take 24 hours), or running it through that ismyblogworking.com. But, its working now!

Thank NAME, for their help.

Let NAME know exactly why they deserved these points.

Gift a custom amount of points.