Questions About Cloudflare Super Bot Fight Mode

I’ve been thinking about testing the new Super Bot Fight Mode that CloudFlare launched a few days ago: Introducing Super Bot Fight Mode

However, there doesn’t seem to be any documentation available about this.

I have a few questions. Anyone else with questions, feel free to chip in on this thread and hopefully we can get them answered by someone from CloudFlare.

  1. What does “Challenge” mean in the settings? Screenshot by Lightshot

Usually it is possible to choose either “Challenge” or “JS Challenge” and the difference between these is massive when it comes to user experience in case of false positives. One is acceptable, the other a deal-breaker if it ends up challenging too many real users.

  1. Is it possible to whitelist IPs, ASNs or user agents to let them get past the bot fight mode?

There have been some reports of good bots being blocked, and requests used to trigger CRON jobs being blocked as well. These not working can really hurt a website.

I think this is a very exciting feature, but blocking and challenging requests can have devastating consequences for a site if something goes wrong. So more information is needed about what this does.

4 Likes

It’s basically the same as the “Challenge” action in firewall rules - which is Captcha challenge.

I read a topic about bypassing bot fight mode using IP access rules, but somehow I can’t find it right now.

1 Like

Super Bot Fight mode, available on the Free/Pro/Business plans, does not have the firewall rules that would allow for whitelisting via IP/ASN/User Agent - those options are only available in Enterprise Bot Management. From https://blog.cloudflare.com/super-bot-fight-mode/:

Unlike Bot Fight Mode, Bot Management is built directly into the Firewall. This means that users can restrict their bot protection to a particular path (like a /login endpoint). Bot Management also includes granular bot scores, which users can pair with other attributes to produce more powerful protection. It even includes Anomaly Detection, which we use to recognize outlier patterns on your site.

2 Likes

That means it will break sites with server-triggered CRON jobs. Various back-end tasks triggered with CRON will stop working, as well as automatic updates which will cause security problems.

Maybe there should be some help docs that explain this since it’s a pretty huge problem that is going to cause problems for a lot of sites.

3 Likes

Unfortunately without some form of whitelisting, Super Bot Fight mode would not be of any use to some folks who will get good automated bots caught in managed challenge/blocks :frowning:

6 Likes

Yup, this hit me yesterday. I was trying to use curl to pull a script from a server and it kept getting called a bot. I thought I’d just whitelist the IP via a firewall rule, but that didn’t work. I ended up having to disable bot protection entirely.

3 Likes

We tested the Super Bot Fight Mode for a few days and it seemed to work very well.
Unfortunately for Pro users without the option to respect our Firewall rules it is basically useless.
Case in point - webhooks from various known services including Github.
We couldn’t even allowlist our own CIDR and our servers couldn’t communicate with each other via HTTP.

9 Likes

We were excited to test Super Bot Fight, but without the ability to at least whitelist IPs / user agents, it’s not useful for us at all. It was blocking RSS feed requests, our own application that hits an API, and ad network scrapers that provide content aware ads.

6 Likes

We are serving API from a subdomain and rightfully so all API traffic went down to zero when BFM was enabled. The same applies to all incoming webhooks from external services.

Very unfortunate, since it makes it pretty much useless.

4 Likes

Same feedback for Cloudflare from here. Excited to try super bot fight mode, but it blocks all webhooks from Sendgrid, Postmark, Stripe, etc. Without the ability to whitelist a user agent it’s hard to see how we could use this.

4 Likes

I would say that path/domain based rules would be much more flexible (UA whitelisting is more fragile and cumbersome). Firewall rules is the most logical place for it, but it is only available on enterprise (not even on business plan).

1 Like

Hmm, even on my Enterprise account, the firewall rules didn’t work. Maybe I’m doing something wrong…?

EDIT: IP Access Rules worked.

Enterprise accounts Bot Management has full CF Firewall control unlike Pro/Biz which do not integrate with CF Firewall so less useful. So with CF Enterprise Bot Management you set your exclusions/whitelisting via CF Firewall rules example

So I can set to challenge, block or simulate/log requests to /login/, /register/, and /contact/ url paths for likely bots with bot score 2 to 29 but exclude IP = 1.1.1.1 and exclude User Agent which contains = Botname and exclude verified Good bots

1 Like

Oh, duh. I forgot we could use the Bot Score directly. I was turning it on via the on/off sliders, which behaves the same in enterprise as it does with free and pro.

+1 all the feedback here. I was excited to try Super Bot Fight Mode but it blocks access to our RSS feeds to send out a daily digest of our blogs via email. This means it’s unusable unless we upgrade to Enterprise, which is far beyond our budget at this time.

Please provide some rules that we can make this workable!

Thank you.

4 Likes

Same here. Awesome feature, very useful, but pretty useless if we can’t whitelist legitimate internal bots.

2 Likes

HI,
I set the bot manager to challenge, but this blocked the IPN of the MercadoPago payment gateway.
I added a rule in the firewall to allow this connection through the MercadoPago user agent but this did not work.
At the moment I have disabled bot blocking until I find a solution.
The bot manager also blocks wordpress cron and other genuine elements coming from the server itself. That is, it also blocks connections from the SAME server IP.
It is impossible to use in these conditions.

2 Likes

is it true as one of my friend said that is useless if we can’t whitelist legitimate internal bots. An you must have to remove column form at the end of Link it redirect to 404 page not found…

https://blog.cloudflare.com/super-bot-fight-mode/ i have removed column from the end.

1 Like

Yeah, this caused some headaches and confusion for me as well.

If allowing an IP in the firewall rules isn’t going to work, this needs to be made much more clear in the user interface or the documentation.

4 Likes

I guess you need to read it again:

This is already clearly written in the documentation.

1 Like