Bots from obvious sources are not being blocked by bot protection mode

Morning Everyone

I am having a small trouble where cloudflare is not blocking traffic from sources that are clearly bots, or jsut random people scrapping web pages (ie: where the user-agent says Friendly_Crawler/Nutch-1.20-SNAPSHOT or FriendlyCrawler or even one ClaudeBot/1.0; [email protected])

I have super bot more enabled and everything.
Is there any setting I may be missing to block these requests, or is the only way to add more rules to the WAF to block these unwanted crawlers?


1 Like


I’ll pay for Pro if Super Bot Fight Mode can be improved to do what it sounds like it should already be doing.

get rid of [email protected] scrapes

Currently I am using WAF Rules to block out unknown crawlers that I see, but its a bit of a whack a mole situation, where they appear as fast as I am blocking them, and I cannot just use a blank rule like any user-agent that has bot in the name, otherwise I will block authentic crawlers, and the maintenance of the list would be hellish

I tried blocking IPS, but they use too many. Ended up using the WAF user agent block for below and put a stop to it.

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; [email protected])

That is exactly what I had to use.
I actually started using a mix of user agent and ANS blocking, since some traffic was 100% just crawlers and bots