Bot fight mode is not able to block a bad bot which is crawling our website

Website, Application, Performance Security
1

Hi There,
Some bad bots hitting our website using common user agents, the requests are likely automated and bot fight mode or built-in bot fight mode are not able to identify them.

Below is the list of user agents that are likely bad bots crawling our website, choking our resources and disrupting our services:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36

These user agents looks legitimate but sending thousands of requests to our website on daily basis from different IP addresses and/or ASNs. We are already putting CF managed challenge for these requests however, they somehow are able to bypass the challenge and crawling website.

Please advice how can we block these type of bad bots under the disguise of common user agents using CF services, we are open to upgrading our plan if that resolved our issue.

Thanks,
Muhammad Hamza

2

Hi there,

Can you share which domain you have an issue with?

If you upgrade to Pro, you should be able to use WAF and security events to see more details of firewall events. (I assume the zone is on Free plan)

3

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.