I briefly enabled Super Bot Fight Mode, but had to disable it almost immediately because it was blocking a lot of legitimate traffic. It certainly caught some rather suspicious requests, but it seems to be rather Western-centric; in particular, it doesn’t seem to be aware of several popular Chinese services.
Here’s a list of bots I noticed it blocking that I would’ve rather it allowed:
- Petal Search by Huawei
- curl, even when coming from a residential network–where’s that “AI” and “ML”?
While I’m not too keen on Semrush crawling my sites, their bot is pretty good about identifying itself, and it claims to honor robots.txt. I don’t see any reason for Super Bot Fight Mode to block bots that both identify themselves and honor robots.txt–and even if they don’t honor robots.txt, they’re trivial to block via the User-Agent header.
What sort of customer is Super Bot Fight Mode aimed at? Technical customers are going to expect it to block only the traffic they can’t trivially block on their own: bots that don’t identify themselves. Less technical customers are going to enable it without understanding that it’s a like trying to perform surgery with a chainsaw and end up preventing their website from appearing in one of the world’s largest search engines and a variety of RSS readers–yikes!