[BUG] Bot fight mode blocks cloudflare observatory

Type of feedback

I don’t know / other

Feedback

Bot fight mode should not block the observatory. It is a good bot. Here is an example ray id: 8b2885d0ddee11a3

Are you on a free or paid plan?

If you’re talking about the regular “Bot fight mode”, that one is only meant to be used temporarily, e.g. during an attack, and to be disabled once the attacks have subsided, or to be disabled once you have adjusted your own WAF to block new traffic patterns, that you find hostile.

See:

I am on the free plan.

I just submitted it to the cloudflare list by using the add a bot link:

https://radar.cloudflare.com/traffic/verified-bots

As mentioned above, that still won’t work with "Bot Fight Mode”.

"Bot Fight Mode” is an all or nothing.

Same goes on for e.g. “I’m Under Attack Mode”, that similarly is an all or nothing.

You will need to use Super Bot Fight Mode on paid plans, if you want to allow (certain) bots to pass through.

The alternative, on the free plan, will be to disable "Bot Fight Mode” , and create your own custom WAF rules, that are matching your desired plans.

| DarkDeviL MVP '24
August 13 |

  • | - |

richard95:

I just submitted it to the cloudflare list by using the add a bot link:

As mentioned above, that still won’t work with "Bot Fight Mode”.

"Bot Fight Mode” is an all or nothing.

Oddly, I am seeing legitimate bots go unblocked. webpagetest.org definitely is unblocked as is pagespeed.web.dev. I suspect search engine spiders are unblocked too as I have yet to find evidence that they are. I could scrutinize the logs more closely to confirm that, but given the other good bots went unblocked, I suspect that it is unblocked too.

Is bot fight mode really all or nothing?

I just checked the logs:

Analytics reports the following visits from known bots in the past 24 hours:

  • Google 31
  • Yandex 9
  • Bing 6
  • applebot 1

Meanwhile, over 2000 requests were challenged by bot fight mode in the same period. >99% had no user agent. Of the remainder, most of them were cloudflare observatory. The remaining 8 were:

  • 2 using the go http client from different IPs in India
  • 2 claiming to be an Intel Mac running OS X 10.15 at what looks like AWS from different IPs
  • 2 instances of the SpeedCurve WebPage Test at different IPs
  • 2 connection attempts from ChromeOS at Google’s cloud, using the same IPv6 address

The analytics data and WAF logs indicate that bot fight mode is not all or nothing. It is intelligently distinguishing between known good bots and potentially malicious unknown bots. Many of the user agent less requests are obvious vulnerability scans trying to access destinations such as /fw.php, /shell.php, various word press administrative pages (which is funny as we do not run word press), etcetera.

Interestingly, I have no clue who would be running SpeedCurve WebPage Test on our site and it could be a bot impersonating another bot.

What I meant with all or nothing, it was that it is either enabled, or disabled.

There is nothing in-between. No kind of granularity, at all.

The issue is, if it ends up on hitting what you call a “known good bot”, such as for example Cloudflare Observatory, then you have no ways to override that, and to allow it through, except from disabling "Bot Fight Mode” completely.

That is the only way, if you wish to allow Cloudflare Observatory to pass through.

I don’t want a way to override it. I want Cloudflare to fix it. This is their bug, not mine.

This is documented behavior:

Once you get to the paid plan level, there’s some granularity (link below), plus the ability to create a bypass rule in the WAF (as described in the above doc).

What I would suggest is to put this in as a Feature Request:

Cloudflare omits bots on their good bot list from being affected from bot fight mode. They need to add their own bot to their own list. Telling me “not being able to configure it” is documented behavior, while true, is beside the point. I don’t want a knob to tune to workaround an incomplete allow list. I want cloudflare to fix their own list. It is ridiculous that they do not allow list their own bot.

By the way, why is the technical term w-h-i-t-e list being censored? I am not writing allow list. I am writing w-h-i-t-e list. I don’t even know what an allow list is. :confused: