robots.txt not working

WordPress has a dynamic robots.txt file which I understand should be overridden if you upload a physical robots.txt file to the server (in the root directory).

I've done that, but when I go to the domain/robots.txt it's still returning the dynamic default WordPress one. Google Webmaster Tools and other online robots.txt checkers are also seeing this.

Can you help me figure out why the file isn't showing? This is a live site, so I'm reticent to do a theme and plugin compatibility test.

  • Adam Czajczyk
    • Support Gorilla

    Hello itsnotrocketsurgery

    I hope you're doing fine today!

    Yes, if a robots.txt file physically exists in a root folder of your WP install (the path where the "wp-config.php" file is located too), that one should be read.

    My first bet on this would be some cache, either on site or on server or both. If you got any caching plugin, clear the cache there but if you got any server-side cache clear it up as well, preferably in that order: clear site cache, clear server-side cache, clear site cache again (just in case).

    If that's not a case or doesn't help, it's also possible that the "robots.txt" file on a server has some syntax errors (so it's ignored and the virtual one comes to play). If that's not this either, then we'll have to investigate it further.

    Best regards,
    Adam

  • itsnotrocketsurgery
    • The Incredible Code Injector

    Hi Adam,

    Thanks for the reply. We do have WP-Rocket installed, but I checked with them directly (and also cleared the site cache anyway) but they confirmed it doesn't cache the robots.txt file or touch it or process it in any way.

    I'm not aware of any server cache configuration (this is cPanel) but I'm pretty sure there's no caching in the way. Thanks for the suggestion.

    So it's entirely possible the robots.txt file has syntax errors. I didn't create the file, it was given to me by a client for their site, and I'm not familiar with it. Is there anything in particular I should be looking out for?

  • Adam Czajczyk
    • Support Gorilla

    Hi itsnotrocketsurgery

    Thank you for your response!

    I'll be glad to "dig deeper" but I'll need a direct access to the site in this case. Could you please provide me with it?

    Note: Don't leave your login details in this ticket.
    Instead, you can send us your details using our contact form https://premium.wpmudev.org/contact/#i-have-a-different-question and the template below:

    Subject: "Attn: Adam Czajczyk"
    - Site login URL
    - WordPress admin username
    - WordPress admin password
    - FTP credentials (host/username/password)
    - cPanel credentials (host/username/password)
    - Folder path to site in question
    - Link back to this thread for reference
    - Any other relevant urls/info

    Once I got credentials I'll check the site and I hope I'll be able to solve that for you :slight_smile:

    Best regards,
    Adam

Thank NAME, for their help.

Let NAME know exactly why they deserved these points.

Gift a custom amount of points.