I guess you need to read it again:
This is already clearly written in the documentation.
CURL (python/perl/generic C HTTP library) is detected through TLS fingerprinting, just give up or disable WAF or make a CFW worker on a diff domain without WAF, and have CURL connect to that, then the CFW worker connects back to CF other zone that traditionally is client eyeballs.
fetch() from chrome or firefox dev console is never caught. NodeJS/Electron/headless browser/webview is caught through the JS challenge. If you up the security even higher, JS challenge turns into a CAPTCHA challenge. before about Nov 2020, or summer 2020, CF offered CAPTCHA challenge with no JS. People (github types) automated hand solving (and AI solving) the captcha no-JS mode, the cat and mouse turned into JS only or JS+captcha and captcha+no JS option is gone . If your customers break captcha+JS, you have a law enforcement problem with “your customers” or a static asset vs dynamic asset vs cloud metal AWS financial problem, not a CF technical problem.
In 2021, any CF “captcha+JS” solving software is closed source and very expensive 0-day bitcoin SaaS. At that point running a bot is very “unprofitable” for the blackhat, since he is paying someone in bitcoin to solve captcha+JS per 30 minutes, per IP, per 1000 http reqs, per whatever. Unless its sneakers or concert tickets or gaming consoles, its unprofitable.
You could also create a static only copy of your website that only bots see and 302 forward on user agent from dynamic->static or static->dynamic (client side JS check if googlebot) and have search engines only index the static half and use
<a> attributes (nofollow?) to stop bots from going from static pages to dynamic.
Verified bots are yandex/google/bing/baidu/naver coming from correct IP ranges. Your basic well known crawlers.
I suspect “super bot mode” is a successor to “API abuse” feature. CF assigns “UIDs” to each client, and remembers page requests in a proprietary secret rolling window (and tossed fast enough to avoid privacy or LE subpoena requests). /login /oauth2 /home → /newest or /inbox. Doing /login → /oauth2 → /change-email is a bot. How did the client magically skip /home or /settings?
Hand coding such logic would be a nightmare. CF can automate it away (but again, this is armchair engineering, I dont work for CF).
Like previous comments, as it is, unfortunately, Super Bot Fight is a bit of a false advantage. It blocks too many services that would otherwise be essential to some sites e.g. RSS feed readers, which are a big deal for dynamic content sites. As soon as I enabled SBFM my users started complaining - and they are falling under both Definitely Automated and Likely Automated.
Also, I was told some services are listed as Known Bots (updown for instance) but still got trapped in Likely Automated and started sending notifications the site was down.
Without a Bypass SBFM option in the firewall rules, using this feature is not a good option for some.
Agreed. There is no point in even offering this feature to anyone but Enterprise customers if they are going to not allow a simple Firewall rule to Bypass the Super Bot Fight Mode with something as simple as a User-Agent.
I have already opened a ticket pointing out where Cloudflare is blocking a HUGE advertising bot used that gets triggered if anyone uses Google Adsense. I have also caught Cloudflare blocking a verified Bing bot used to generated snapshots of the website.
This is poorly implemented and an insult to give a feature like this to paying customers with quality control like this.
Then keep it an Enterprise only feature because it is insulting giving paying customers a feature that is essentially harmful to enable to the majority of people who run websites.
It looks like even Let’s Encrypt is being blocked by this. Like, seriously? This could bring down websites that enable this without thinking about how terribly Cloudflare vetted this feature.
Here is SBFM blocking a Cloudflare feature, Real AMP URL: https://twitter.com/brentwilson85/status/1379087591526531072
Ouch @bsolomon !
Ben is OOO, but I checked with his team for some additional feedback. Long story short, this feature is intended for people who want an easy way to upgrade their security against bots without a lot of fine tuning. Users who want/need a more granular approach are probably going to want to go with a different solution.
Also, legitimate bots should be allowed as long as you are set to allow “verified bots”. So we are looking into the possibility that some unwanted blocks are occurring.
Not trying to be rude but this kind of renders the feature useless. We are in an era in which we are “APifying” everything and almost any website has some sort of consumer in the sense of cronjob or similars, not being able to whitelist these cases is rather problematic.
I get you want people to pay $4,000 a month or more for the option of a firewall rule to bypass SBFM via User-Agent, but I am not and this feature is absolutely dangerous to enable without something as simple as that. We can’t trust you to be on top of adding every new bot that is friendly to your service in a timely manner.
Services that we intentionally sign up for, for example, like Flipboard, are also automatically blocked, blocking our news articles from ever appearing on that service. The list goes on.
On top of all of this, your support is asking customers to fill out a form meant for the bot owners as a solution. This is not feasible.
Thanks for contacting Cloudflare support.
To be added to the Cloudflare allowlist, please submit this online application.
Please let us know if you have any further questions or issues by replying to this e-mail or ticket.
Agreed, whilst the feature is cool. Not giving us the ability to fine tune / customise what is whitelisted from the firewall rules section basically makes this feature an arbitrary firewall that can silently kill large amounts of legitimate traffic to your site overnight. When we enabled the feature, wierd issues started happening… ajax requests failing / users being challenged for accessing from corporate / education ASN’s.
I came here today to ask if Cloudflare is aware that both bot fight and super bot fight are close to useless to nearly anyone other than enterprise customers since there is no way to whitelist needed services.
I see that you are and have decided that it’s ok to offer this feature to people who are not going to understand that it’s very likely to break functionality they are currently using in ways that is not transparent to them. I think you should put some safety text above the toggle switch to warn people who may not understand what BFM and SBFM can and will break.
I just enabled Super Bot Fight (Pro level). My first report is this:
These numbers seem insane to me, especially the verified bot number. And this is only for past 24 hours.
What’s your thoughts?
My thoughts are you need to pay attention to your Firewall log so you make sure this isn’t rejecting bots from your site that you need. Especially if you run any 3rd party advertising.
Support is still providing the solution as simply “Do not use it” and ignoring how dangerous this feature is as it sits. I can’t come up with a website on a pro or business plan that can enable this feature without it harming their site. Literally, you would have to have a static website with no features that include automation to benefit from this feature.
While I am pretty sure it was not the intention, but it does look like a feature aiming to upsell enterprise deals in its current scope.
It’s very hard to imagine a modern website beyond a very simple static one that doesn’t deliver data to legitimate programmatic consumption (or consume webhooks from external sources).
Do you have a suggestion for what support could do differently in its response? I’d be happy to pass along proposed language changes to the team.