Notable tools and services blocked by Super Bot Fight Mode

I briefly enabled Super Bot Fight Mode, but had to disable it almost immediately because it was blocking a lot of legitimate traffic. It certainly caught some rather suspicious requests, but it seems to be rather Western-centric; in particular, it doesn’t seem to be aware of several popular Chinese services.

Here’s a list of bots I noticed it blocking that I would’ve rather it allowed:

  • Petal Search by Huawei
  • Sogou
  • Inoreader
  • Scoop.it
  • curl, even when coming from a residential network–where’s that “AI” and “ML”?
  • FeedWind
  • FeedWordPress
  • Semrush

While I’m not too keen on Semrush crawling my sites, their bot is pretty good about identifying itself, and it claims to honor robots.txt. I don’t see any reason for Super Bot Fight Mode to block bots that both identify themselves and honor robots.txt–and even if they don’t honor robots.txt, they’re trivial to block via the User-Agent header.

What sort of customer is Super Bot Fight Mode aimed at? Technical customers are going to expect it to block only the traffic they can’t trivially block on their own: bots that don’t identify themselves. Less technical customers are going to enable it without understanding that it’s a like trying to perform surgery with a chainsaw and end up preventing their website from appearing in one of the world’s largest search engines and a variety of RSS readers–yikes!

1 Like

I unfortunately discovered it blocks UptimeRobot. It seems to be an all-or-nothing approach for now.

1 Like

Yup, it seems to block more or less everything except the biggest Western search engines, plus maybe Yandex and Baidu (I didn’t check to ensure they went through, but I didn’t see them in the firewall logs for the minute or so I had Super Bot Fight Mode enabled). I’m not really sure what the use case for that would be. Personal websites that don’t want anyone except a handful of people reading their content?

All Google Feedburner, Zapier and any other RSS Feed reader…

Which is a bummer, as I really wanted to use Super Bot Fight Mode but blocking RSS is a big no-no for dynamic sites - we have quite an audience that rely on RSS to come back.

Ideally we could add a Firewall rule with Bypass Super Bot Fight Mode but there is no such option, I am told by support.

2 Likes

We’re also having an issue with Google Feedburner being blocked in Super Bot Fight Mode.

Feedburner should be whitelisted globally by CloudFlare or they should provide users a way to allow it ourselves. IP ranges and Page Rules don’t seem to help.