Field "Known Bots" (cf.client.bot) not filtering hits correctly

[alexgarciab]

What is the name of the domain?

Example Domain

What is the issue you’re encountering

Field “Known Bots” (cf.client.bot) not filtering hits correctly

What steps have you taken to resolve the issue?

Deployed a Custom Rule such as: (not cf.client.bot and ip.geoip.asnum in {8075})

What is the current SSL/TLS setting?

Full (strict)

What are the steps to reproduce the issue?

The Custom Rule filter associated with “cf.client.bot” is incorrectly tagging bad hits that should be blocked.
I am getting malicious hits from ASN 8075 and the filter is failing to apply the logic.
Here’s an example of one of the multiple requests that wen through and it shouldn’t.

Screenshot of the error

[alexgarciab]

Just for reference, the offender IPs come from Microsoft ASN - The example IP is: 52.169.233.48 which is clearly and abuse IP: https://www.abuseipdb.com/check/52.169.233.48

[paul32]

We started by blocking all of AS8075 from ireland as it was 100% attack attempts then just scaled up to blocking AS8075 as its responsible for about 80%+ of the script kiddie attack attempts on our sites

[alexgarciab]

But you can’t just block the entire ASN, because there are legitimate bot visits coming from Microsoft ASN such as BingBot visits or Microsoft Ads visits.

This is challenging because attackers are using an ASN that seems trustworthy (Microsoft) and has good bots, in a deceptive way.

Let’s see if Cloudflare support can bring some light to it.

[paul32]

We can and do block the entire ASN - bingbot doesnt follow robots.txt so is blocked

The ASN is THE major source of attacks on our sites so gets blocked

[alexgarciab]

Similar issues for reference:

WAF Rules - Lists not working?

What is the name of the domain? any What is the error number? %s function forbidden fields, at index: 0 What is the error message? %s function forbidden fields, at index: 0 What is the issue you’re encountering WAF rule which has been in place for years now changed from showing in Expression Builder to only showing in text box, trying to save give “unsupported” message What steps have you taken to resolve the issue? Rewrite WAF rule step by step:: ip.geoip.asnum in {15169 396982} is OK ad…

Hacked Microsoft Network bypassing Bot fight mode, Managed Challenge

What is the name of the domain? N/A What is the error number? Ray ID 8d172a2bfda1be2c What is the error message? Action taken: Managed Challenge What is the issue you’re encountering Hacked Microsoft Server attacking my website. Bypassing Bot fight mode and is managing the challenge. I Do Not run WordPress. What steps have you taken to resolve the issue? Blocking attacking IP’s, blocking ASN’s and country. The attacker is bypassing Cloudflare WAF and bypassing bot fight mode. Was the site w…

Country block and receiving large amount of requests 2k+, attack?

What is the name of the domain? What is the issue you’re encountering Large amount of GET requests “Managed Challenges” What steps have you taken to resolve the issue? Large requests 2k+ daily from specific country, even after adding WAF block rule. WAF does not appear to be working. Example of Get requests IP address 13.79.229.207 ASN AS8075 MICROSOFT-CORP-MSN-AS-BLOCK Country Ireland User agent Empty user agent HTTP Version HTTP/1.1 Method GET Host encryptifyme.com Was the site w…

Bot fight mode blocking Bing Bot

What is the name of the domain? example.com What is the error number? None What is the error message? None What is the issue you’re encountering AS8075 is blocked by bot fight mode What steps have you taken to resolve the issue? I tried to add the ASN to a WAF rule but it seems bot fight mode takes precedence over WAF rules: (cf.verified_bot_category eq “AI Crawler” and not ip.geoip.asnum in {15169 139190 8075 396982 20940 13335 4113 19551 36040 32934 63293 30148 16509 5466 15502 396982 807…

Microsoft 'good' bots hammering away

I have a security/waf rule that allows only Microsoft ASN’s ‘good’ bots through: (ip.geoip.asnum eq 8075 and not cf.client.bot) which is set to ‘challenge’ and has worked really well for years until about a week ago where the class C 52.169.23.0/24 is allowed by this rule (assumed deemed ‘good bot’ by cloudflare) and hammers my site all day long with malicious intentions. How can I get cloudflare to review this class C and remove them from their ‘good bot’ list? J

Cloudflare WAF false positive: BingBot

Auto-closing topics too quickly really is an issue here. There are dozens of auto-closed issues about this topic, while it was never resolved. I just link one here, and create this as follow-up, in the hope it is not again auto-closed: Cloudflare blocking bingbot crawl @tye730 @fritex pinging you here, as you are probably interested. Today I switched to the new managed WAF rules, and watched the event log. Cloudflare’s own ruleset triggered a block of the following request: { "action": "blo…

[paul32]

The first one in your list is an issue I raised and has nothing to do with AS8075, it was an upgrade issue

[alexgarciab]

I believe something is not working with Cloudflare Custom Rules.

I have deployed these 2 rules:

(not cf.client.bot and ip.geoip.asnum in {8075}) > BLOCK
(cf.client.bot and http.user_agent eq "") > BLOCK

So basically this should be the logic:

1. Mutually Exclusive Conditions: The core condition differentiating the two rules is cf.client.bot.

• Rule A (Original) requires not cf.client.bot (i.e., cf.client.bot must be FALSE).

• Rule B (New) requires cf.client.bot (i.e., cf.client.bot must be TRUE).

• A single request can only have cf.client.bot be either TRUE or FALSE, never both. Therefore, a single request can never match both rules.

2. Request Evaluation:

• Your Target Request: cf.client.bot is TRUE, http.user_agent is “”.

  • It will fail Rule A (not cf.client.bot is FALSE).

  • It will match Rule B (cf.client.bot is TRUE and http.user_agent is “”).

  • Since it only matches Rule B, it will be blocked by Rule B regardless of whether Rule A or Rule B is checked first.

• Other Request Type 1: cf.client.bot is FALSE, ip.geoip.asnum is in the list.

  • It will match Rule A.

  • It will fail Rule B.

  • It will be blocked by Rule A, regardless of order.

• Other Request Type 2: cf.client.bot is TRUE, http.user_agent is not empty (e.g., “Googlebot”).

  • It will fail Rule A.

  • It will fail Rule B.

  • It will not be blocked by either rule, regardless of order.

But still these requests are not being BLOCKED by the Custom Rules:

[paul32]

The IP address 40.69.216.239 is not a verified bingbot so should, as you say, fail test 1

So looks like Cloudflare saying cf.client.bot is TRUE for that IP address as it should be failing BOTH tests

Either that IP is failing the not cf.client.bot test
or its failing the http.user_agent eq “” test

I have a rule that uses the http.user_agent eq “” test so reckon that works - so it looks like the IP is getting cf.client.bot = TRUE when it shouldnt

Or - do you have any other rules or any “allow” IP ranges?

[sammyt]

I’ve also been having problems trying to block similar traffic.
I have a (http.user_agent eq "") rule but it only appears to block about half of the matching requests.

[alexgarciab]

Yes, I am also seeing the exact same behavior. It’s like half of the visits don’t get blocked by the SAME Custom Rule. Here’s a graph that shows this:

@sjr any clue why this is happening?

[alexgarciab]

I have fixed the issue by implementing these changes:

• Disable both fight mode
• Move Redirect Rule (WWW to non-WWW) from Cloudflare to nginx rule
• Changed “ip.geoip.asnum” to “ip.src.asnum”

And the explanation is simple. Bot fight mode in the free Cloudflare tier is very bad. If you have proper Custom Rules in the Firewall, these are much more effective than their “Bot fight mode”. Redirect Rules get evaluated before Custom Rules, so Cloudflare was redirecting WWW to non-WWW and not applying the Firewall Custom Rules to those hits. I moved the WWW redirect to my own server instead of Cloudflare. And finally, the “ip.geoip.asnum” for ASN blocking is deprecated. The correct expression is “ip.src.asnum”.

[sammyt]

Thank you so much! Turning off Bot Fight Mode and Block AI Bots made a drastic improvement.