Robots.txt
I can't find the robots.txt file. It's causing the problem below and needs to be removed. Any help is appreciated.
erealestate.com is currently blocked by a robots.txt file that has been uploaded to the site: http://erealestate.com/robots.txt. Should the file be stricken or adapted to allow the crawler ia_archiver, crawling for the Wayback Machine can continue. Until then, the presence of the site is considered an express request to exclude the specified domain from the Wayback Machine.
Everyday at WPMU DEV we help hundreds of WP Users. Become a member today to:
- Download 250+ Premium Plugins and Themes
- Get unlimited support from WordPress Experts
- Get help with your WordPress or BuddyPress projects
Help & Support from WPMU DEV members and staff
Keeper of the Dark Chocolate
Your robots.txt file is fine. WordPress now generates it on the fly.
Actually it looks like you have the blog in question set to not be indexed by search engines as this is appearing within the blog's outputted source.
<meta name='robots' content='noindex,nofollow' />
Take a look at the Privacy settings and make sure you;ve set it to be indexed by search engines. If it is, the line above maybe hardcoded into the theme's header.php file and you;ll have to remove it.
I have a feeling that it's hardcoded as if you had set your blog to not be indexed, the robots.txt file would also reflect that and it's not.
Hope this helps,
-drmike
edit: Oh, and I;ve moved this thread to the Discussion subforum. As per the forum descriptions on the main page, Andrew would rather have the Plugins Support subforum only for the plugins available here.
Your robots.txt file is fine. WordPress now generates it on the fly.
@drmike Does that mean we no longer use James' robotstxt WPMU plugin?
http://wpmu.org/wpmu-robotstxt-globally/
Get personal, comprehensive and timely support and assistance
from WordPress Experts
Keeper of the Dark Chocolate
Um, no disrespect to him but James' example is rather lacking and incorrect. It has a couple of dupes and I for one would rather include the mu-plugins subdirectory as well.
Heck, if you;re going to do that, I'd just place a robots.txt file within root and put in there what you want. I do agree with James though that 99% of your endusers aren;t going to understand what a robots.txt file is anyway unless you;ve got a specific niche that you cater towards.
The kbrobots plugin he links to is another option and what we use on our installs. Do a search of the forums here though for kbrobots as I had a security concern about it that really never got answered. (I;ve got 2 minutes of net time left.)
don't mean to dig up an old thread, but I am having this same issue.
Google says that it is being blocked by a robots.txt file
http://daumanelectric.net/robots.txt
User-agent: *
Disallow:
But I have the privacy settings as index me.
Any thoughts?
Can you check with an FTP program and see if you actually have a robots.txt file in the root of your WP site as an actual file existing will override the WP created version.
The privacy settings you mention are the ones in Settings -> Privacy, correct?
There is no robots.txt file in the root. If added would this effect all sites? What i one needed to be hidden for whatever reason?
Yes that is what I mean by privacy settings.
Get personal, comprehensive and timely support and assistance
from WordPress Experts
Yes, a robots.txt in the root would affect all sites.
Sorry, just looking at the code in WP - that robots.txt seems to actually be correct as far as WP is concerned.
If it looked like this:
User-agent: *
Disallow: /
Then it would block access.
Can someone more up on robots.txt confirm this (or not) the WP function is in wp-includes/functions.php - line 1779 function do_robots()
It says allow now, I guess you fixed?
Hi there- it looks like I'm having the same problem at http://gbiportal.net . I did have my privacy settings set to hide from search engines, but I've switched it to be visible to everyone. However, I'm still getting "blocked by robots.txt" message from Google. I have tried creating a new robot.txt file, deleting any that I see, but still the same thing.
I am now not seeing any robot.txt files (from my ftp) but can still see User-agent: *
Disallow:
at http://gbiportal.net/robots.txt
What's going on?
Where should I look for the : <meta name='robots' content='noindex,nofollow' /> mentioned in DrMike's first response?
I'm using BP-Daily theme, in case it is relevant.
Thanks,
Laurie
So, I never figured out what was wrong. I ended up finding a plugin that wrote a new robots.txt file for me (although why the one I was trying wasn't working is beyond me). That still didn't work. BUT, I then ran the site through ismyblogworking.com - this verified that the robot.txt file was fine. Soon after that, I checked with google again, and this time it found it worked fine! Not sure what was the winner - new plugin's robot file, waiting several hours since I reset the privacy settings (somewhere I saw it could take 24 hours), or running it through that ismyblogworking.com. But, its working now!
Get personal, comprehensive and timely support and assistance
from WordPress Experts
Hiya IntegraLLC,
Glad you got this working! :D
Especially, thanks for posting your results. I'm sure that'll be helpful to others
