User Agent Blocking Partial Failure

Website, Application, Performance
1

What is the name of the domain?

What is the issue you’re encountering

User Agent Blocking Partial Failure

What steps have you taken to resolve the issue?

Currently using WAF Custom Rules with this expression:

(lower(http.user_agent) contains “bytespider”) or (lower(http.user_agent) contains “amazonbot”) or (lower(http.user_agent) contains “meta-externalagent”) or (lower(http.user_agent) contains “facebookexternalhit”) or (lower(http.user_agent) contains “semrush”) or (lower(http.user_agent) contains “claudebot”) or (lower(http.user_agent) contains “ahrefsbot”) or (lower(http.user_agent) contains “dotbot”)

Cloudflare blocks all requests matching those user agents coming from IPv4 addresses. However, looking at our server logs, we can see requests that come from IPv4-mapped IPv6 addresses are not blocked. Here’s an example:

::ffff:54.36.149.20 | Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)

2

Here’s an example of two requests from AhrefsBot within 5 seconds of each other:

8:32:20 PM - 54.36.148.150 | Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/) - blocked, shown under cloudflare firewall events, not seen in our server logs
8:32:25 PM - ::ffff:54.36.149.20 | Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/) - NOT blocked, not shown under cloudflare firewall events, visible in our server logs

3

Also just wanted to clairify, this is happening with multiple bots, not just ahrefs.

4

We are still experiencing this issue. Anyone else?

5

Wonder if they’re hitting the same URL of the domain, or rather the main server hostname available over the HTTP(S) port? :thinking:

6

Both requests are to the same https URL (e.g., mywebsite.com - This website is for sale! - mywebsite Resources and Information.). It’s just that cloudflare’s firewall seems to ignore the ipv4-mapped ipv6 traffic.

7

Still having this issue… can anyone replicate?

8

AS far as I am blocking the ASNs, therefore it’s blocked at the IP Access Rules via multiple ASNs. Will give it a try with another CF account via WAF Rule to test out the UA for IPv4/IPv6 if it catches it.

9

This topic was automatically closed after 15 days. New replies are no longer allowed.