Blocking aggressive crawlers / bots

I have access to WAF but it is not blocking bad bots very well

For example today I had 5786 requests from this ip address in a short period of time today.

Surely WAF should be blocking this? Are there any additional settings I can use to block these kinds of bots automatically because this seems to be happening fortnight to me

Unless the IP in question is hitting enough endpoints and/or rules, it may not automatically trigger a block. One suggestion, have a look at Cloudflare’s Rate Limiting by going to Dashboard > Firewall > Tools
to block after x amount of requests.

Also, create some free Firewall Rules! Using the Community search :search:, find recommended rules based on your platform.


Thanks, have added quite a few rules to my firewall but none seem to trigger these bots. I did see rate limiting but its a bit above my budget. I

In the last 30 days I had 16,484,985 requests. I am sure some of these requests would get rate limited so this number in theory should be lower however based on my calculations

16,484,985/10000*.1 = $164.84 per month eek!

I think rate limiting only count against requests hitting your origin(as in most sites most of the requests are cached)

edit: I have a site with 30M requests and I see they are charging me 0 for rate limiting… looks like it broken

Thanks for the info boynet2, I will try to find out from support an estimate and come back with what they say.

Edit: Just heard back from support, my math was off its $0.05 per 10,000 requests. However I have been told that it will cost $82.40 based on 16,484,985 requests. Still quite pricey to be honest

1 Like

This topic was automatically closed after 30 days. New replies are no longer allowed.