Frequent 404 lockouts

I have a weird issue with Defender that I was hoping you could help shed some light onto.

My client as well as some of his customers keep locking themselves out of the website due to 404 requests. Checking the logs though, I see that these 404 requests appear to be to images or pages that don't exist, but they could be related to demo data that was in the theme initially

I can't see where I am making requests for them on the website's pages, and I have performed a couple of broken link checks both with third-party apps and through the Console with no result — there seem to be no broken links on the website but Defender somehow finds them.

  • jnkfrancis

    Hi Kronikon Design,

    Thanks for reaching out with your concerns over multiple 404 Lockouts.

    I think it would be helpful if I started by explaining a little bit how Defender works in regards to 404 errors and what the data you are seeing in your summaries and logs means. To start, Defender will block an IP when it makes multiple requests for a page that is not on your server. This can be from innocent mistakes, old expired links, or from a spider just crawling your site and indexing it. If an IP makes multiple requests for files you don't have, in a short amount of time, Defender considers that suspicious and will lock out that IP from accessing your site for a set amount of time. Or in the case of particularly abusive IP's it will block them permanently.

    So what you are seeing in your logs are records of outside sources (legit users, bots, spiders etc) attempting to access files you don't have. It doesn't mean necessarily that YOU have a bad link, but rather that there are some bad links out there to your site. You can access your logs and see exactly who is looking for what (see screenshot 1). If you find that the IP is associated with a known user, you can whitelist that IP so defender will ignore their 404 errors. However if you see an IP that you don't know and you do some research and find they are a spammer or a known abuser, then you can permanently ban their IP. I would use this option cautiously as you don't want mistakenly ban a potential customer or a "good" bot from Google.

    So going forward, I took a look at your site, and I noticed that some of the settings for Defender's 404 Detection don't look right. Specifically I would suggest that you go to Defender Pro > IP Lockouts and select "404 Detection". On that page you should have some values in the Lockout Threshold and Lockout Time sections. We recommend a Lockout threshold of 20, 404 Errors in 300 seconds and setting the Lockout time to 300 seconds as well.

    What that means is: If an IP makes 20 requests in 5 minutes of pages that don't exist, Defender will block that IP for 5 minutes.

    Finally, as an additional measure to prevent your logged in users from being locked out because of 404 Errors, I would recommend that you uncheck the checkbox Monitor 404s from logged in users.

    All of this is illustrated in (screenshot 2).

    Finally it's always best to be sure that your plugins stay up to date, and I noticed that Defender needs to be updated (We made some updates to IP Lockouts). So when you get a chance, be sure to backup your site and make those updates to keep everything running smooth.

    I hope that clears things up and if you have more questions please feel free to ask.

    Thanks!

    Jeremy

  • Kronikon Design & Development

    Hi Jeremy,

    Thanks for the explanatory email. I understand how Defender and 404 lockouts work. What I don't understand is why there are so many requests for these non-existent files on my site. If you take a look, you will see that most of them are interior design-related; that makes me think that they are related to the theme's demo content.

    Since this is a team of WP experts, I was actually looking for someone to shed some light into what could be happening here. Could these links be somewhere in the theme files? Something else?

    Thanks,

    Thalia

  • jnkfrancis

    Hi Thalia,

    If you could re-enable the Support Access I would be happy to take another look, but what Defender is logging is outside requests that resulted in a 404 error. So If, for example, I went to your site and tried to access https://thetributeconnection.com/jeremy-page it will return a 404 Error and log that attempt.

    So what that can mean is that if someone is following an old link perhaps from your placeholder content that happened to be indexed, you would get that result. The other possibility could be someone sniffing around looking for vulnerable files.

    So it is unlikely that there is anything wrong on your end, but that you are getting odd requests from old links or someone snooping.

    If you want to track down those bad links, you can use Google Analytics, and Google Webmaster Tools to find any old links and fix them. Here are a few guides on how to use Google Analytics and Webmaster Tools to monitor your backlinks:
    https://blog.pagezii.com/how-to-check-backlinks-in-google-analytics/
    https://www.pageonepower.com/linkarati/monitor-backlinks-google-webmaster-tools-negative-seo

    You can use the Disavow backlinks feature if they are spam or with a 301 Redirect if they are legitimate.

    Here is a handy guide from Google on using Disavow backlinks: https://support.google.com/webmasters/answer/2648487?hl=en

    If you need to do 301 Redirects SmartCrawl can do that for you and we have a great guide here:
    https://premium.wpmudev.org/docs/wpmu-dev-plugins/smartcrawl/#Chapter5Redirect

    And here's a great article on the subject with some other suggestions: https://premium.wpmudev.org/blog/wordpress-redirection-plugins/

    So if you can re-enable support access I will be happy to take a look and while I do that you can look at those resources and see if they are helpful.

    Jeremy

Thank NAME, for their help.

Let NAME know exactly why they deserved these points.

Gift a custom amount of points.