Bots from obvious sources are not being blocked by bot protection mode

Website, Application, Performance Security
1

Morning Everyone

I am having a small trouble where cloudflare is not blocking traffic from sources that are clearly bots, or jsut random people scrapping web pages (ie: where the user-agent says Friendly_Crawler/Nutch-1.20-SNAPSHOT or FriendlyCrawler or even one ClaudeBot/1.0; [email protected])

I have super bot more enabled and everything.
Is there any setting I may be missing to block these requests, or is the only way to add more rules to the WAF to block these unwanted crawlers?

Regards

1 Like
2

Ditto!

I’ll pay for Pro if Super Bot Fight Mode can be improved to do what it sounds like it should already be doing.

3

get rid of [email protected] scrapes

4

Currently I am using WAF Rules to block out unknown crawlers that I see, but its a bit of a whack a mole situation, where they appear as fast as I am blocking them, and I cannot just use a blank rule like any user-agent that has bot in the name, otherwise I will block authentic crawlers, and the maintenance of the list would be hellish

5

I tried blocking IPS, but they use too many. Ended up using the WAF user agent block for below and put a stop to it.

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; [email protected])

6

That is exactly what I had to use.
I actually started using a mix of user agent and ANS blocking, since some traffic was 100% just crawlers and bots

7

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.