Robots.txt
I can't find the robots.txt file. It's causing the problem below and needs to be removed. Any help is appreciated.
erealestate.com is currently blocked by a robots.txt file that has been uploaded to the site: http://erealestate.com/robots.txt. Should the file be stricken or adapted to allow the crawler ia_archiver, crawling for the Wayback Machine can continue. Until then, the presence of the site is considered an express request to exclude the specified domain from the Wayback Machine.
Help & Support from WPMU Dev members and staff
Your robots.txt file is fine. WordPress now generates it on the fly.
Actually it looks like you have the blog in question set to not be indexed by search engines as this is appearing within the blog's outputted source.
<meta name='robots' content='noindex,nofollow' />
Take a look at the Privacy settings and make sure you;ve set it to be indexed by search engines. If it is, the line above maybe hardcoded into the theme's header.php file and you;ll have to remove it.
I have a feeling that it's hardcoded as if you had set your blog to not be indexed, the robots.txt file would also reflect that and it's not.
Hope this helps,
-drmike
edit: Oh, and I;ve moved this thread to the Discussion subforum. As per the forum descriptions on the main page, Andrew would rather have the Plugins Support subforum only for the plugins available here.
Your robots.txt file is fine. WordPress now generates it on the fly.
@drmike Does that mean we no longer use James' robotstxt WPMU plugin?
http://wpmu.org/wpmu-robotstxt-globally/
Um, no disrespect to him but James' example is rather lacking and incorrect. It has a couple of dupes and I for one would rather include the mu-plugins subdirectory as well.
Heck, if you;re going to do that, I'd just place a robots.txt file within root and put in there what you want. I do agree with James though that 99% of your endusers aren;t going to understand what a robots.txt file is anyway unless you;ve got a specific niche that you cater towards.
The kbrobots plugin he links to is another option and what we use on our installs. Do a search of the forums here though for kbrobots as I had a security concern about it that really never got answered. (I;ve got 2 minutes of net time left.)