Defender Useragent Based Whitelisting

We have an issue with Defender IP banning bots from Search Engines and SEO crawlers. The IP addresses change for these commonly, so it's not always possible to whitelist them via IP.

I thought a good idea would be to whitelist by user-agent? as Most of our crawling software allows us to specify this as an entirely custom string.

  • Kasia Swiderska
    • Support nomad

    Hello James,

    Would you mind allowing support access so we can have a closer look why Defender is blocking those IPs? Defender blocks IPs when there is failed login attempt and that hardly case with crawlers. Other possibility is that they are blocked because of the 404 errors - then it should be investigated and see what can be done.
    To enable support access you can follow this guide here:
    https://premium.wpmudev.org/docs/getting-started/getting-support/#chapter-5

    I moved your thread to support forum as it about one of our plugins.

    I'm also checking with developers if we have any plans to include white lists by user agents. As soon they will provide feedback I will let you know.

    kind regards,
    Kasia

  • James
    • WPMU DEV Initiate

    You're correct in mentioning that they are getting locked out because of 404's .

    But, ultimately, part of our SEO work is knowing which accessible URLs lead to a 404 - so I'd want to let a crawler hit these, no matter how many there are. The defender system isn't broken in any way, it's behaving as expected, which is why I put this in feedback and ideas and not support.

    I just wanted to suggest being able to whitelist based on user agent as well as IP to allow crawlers (and any other services) that change IP often to get the access they need.

  • Kasia Swiderska
    • Support nomad

    Hello James,

    The defender system isn't broken in any way, it's behaving as expected, which is why I put this in feedback and ideas and not support.

    Oh, I'm sorry - it was posted in Members forum in Web Development, I thought it was by mistake.
    In this case I will move your thread to Feature & Feedback forum so it can be easily found. In the meantime I dropped a line to developers about your idea and seems that they are already discussing this.

    kind regards,
    Kasia

Thank NAME, for their help.

Let NAME know exactly why they deserved these points.

Gift a custom amount of points.