Bot traffic bypassing Super Bot Fight Mode

I’m seeing bot traffic bypassing Super Bot Fight Mode. Specifically bot traffic from
yournewsecretweapon[.]com

Coincidentally, this bot is slipping past captcha too, which makes me curious. What are your thoughts on why this particular bot is getting through Cloudflare’s bot fight mode?

Do you see the bot even being challenged in your logs? To my mind, it’s infrequent to see bots pass captchas.

Sadly Super Bot Fight Mode is just plain right bad. Any webmaster that knows anything will tell you this, let alone the number of MVPs that have said similar things if not outright bashed it. People have also been asking for better editing of rules and whitelists etc., but nothing has occurred yet.

Like many here, use custom rules. Using the search function, you will find excellent examples and information. Some people here are very passionate about it.

Some examples that will net you more bot control are the following.

Ban TOR

Ban Outdated Internet Browser user-agents - 3 years and onwards. You will see how many use strings from 5 to 8 years ago and be shocked. This is a big one I found in reducing bots big time. Most are too stupid to make fake user agents.

Ban anyone not using a known HTTP version, allow only HTTP 1.0, 1.1, 1.2, 2 and 3, etc.

Ban requests that are not needed, aka options, delete unless you need them. For example, WordPress users are only required to get, push and head.

Challenge or ban the top 10 worse asns and countries unless your customers / good traffic for AdSense comes from those countries. Spamhaus is your source = Spamhaus: The Top 10 World's Worst Spam Support ISPs.

Ban ASN’s that repetitively hit firewalls, wafs, and downright just malicious.

Challenge /Ban countries that you don’t cater to, alternatively challenge/ban the entire continent.
For example, your blog is about something dedicated in Australia, where all the countries in Africa really wouldn’t care, and most cannot even read the language or ling. Still, you get 30% of your traffic from Africa that doesn’t seem right. Instant challenge in my mind to all and see the bots get squatted.

Ban anyone with a threat score greater than 5, lower as needed, but test with JS / HCAPTCHA first if you go low.

These are just a few examples where you can destroy the bots, leaving the super bot mode as, yeah, I won’t say anymore. Also, many more examples in the forums.

Ensure to allow all known CF bots before any of the rules, and this will bypass and help reduce any good bot false positives.

Don’t get me wrong, CF can handle bots, but you need to pay an enterprise service to have any real control.

3 Likes

I realise that Tor can be an unending stream of garbage. But there are plenty of people in situations where Tor is an essential tool. Try and find a balance between the two. If Tor traffic does not cause you any trouble, give the real users in there some slack.

It is something of a blunt instrument. Hopefully future versions will give some granular control.

2 Likes

I wouldn’t say the protection itself is the problem, but rather than a crucial step was missed in the design phase.
Not having any way to whitelist bots or ips properly definitely kills the feature purpose for most customers.

Either way, I don’t think any of the available features are appropriately tuned to block complex bots. It would be best if you had the enterprise package to do that.

This shows that most detections are achieved through ML inference, which is expensive and highly unlikely to reach the public.
For example, our company offers a service that at some point requires machine learning operations in real-time; at peak hours, we need to handle 12000 rps, which costs us approximately $20-25 per hour; that’s a lot.
If we consider that pricing and ignore the rest of the costs (bandwidth, WAF…), those customers would burn their entire monthly quota within 1 hour, depending on the traffic.

Edit: Regarding Captcha, the only way to make it harder for bots to solve hCatpcha would be adding adversary attacks and other tricks that are present in ReCaptcha.
This might make the user experience slightly worse, especially if their bot score is high, however, when you want to fight bots that are capable of solving captchas, you need to assume that some unfortunate people are going to pay for it.

This is what Cloudflare wants you to subscribe. If they add the whitelisting function for Super Bot Fight Mode, then it will make Enterprise-grade Bot Management much harder to sell.

3 Likes

I don’t think it’d be much harder. Enterprise Bot Management is still far more granular than self-service plans. A simple Allow List for bot fighting isn’t going to change that.

2 Likes

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.