Rate limiting on a specific ASN: how to do it?

Website, Application, Performance
1

What is the name of the domain?

N/A

What is the issue you’re encountering

N/A

What are the steps to reproduce the issue?

Hello everyone, there’s the ASN ALIBABA-CN-NET with number 45102 that’s been hitting my robots.txt file non-stop for days. It’s sending about 120 hits per hour. Since this is completely unjustifiable and has become a real nuisance, I’d like to take action to limit these requests.

However, I see that rate limits can’t be applied per ASN. Moreover, the limit I can set per IP is quite high, which risks affecting legitimate ASNs, such as Google, which makes quick bursts of hits in a very short period. What can I do? Ideally, I’d like to implement a rate limit per ASN.

I also considered blocking the ASN directly via WAF, but that feels like overkill…

I’m on a Pro plan.

Thanks in advance!

2

That’s an Enterprise feature (with Advanced Rate Limiting addon).

See: https://developers.cloudflare.com/waf/rate-limiting-rules/#availability

1 Like
3

Make a conditional rule that is exempting Verified bots from the blocking rule?

Is it because you’re seeing an overload on your server, or what exactly is the reason for the wish to apply rate limiting in this case?

Are you performing regular changes to your “robots.txt” file?

I’m thinking, … could applying a cache for the “robots.txt” file at the edge, for an hour or two, or even longer, be combatting the issue well enough?

Together with e.g. Tiered Cache · Cloudflare Cache (CDN) docs, it should be able to provide a great layer of defence.

4

Honestly, the issue isn’t the overload. The resource is cached. It’s just that I don’t think it’s right to keep fetching that resource, as it skews my extremely useful statistics. Do you see what I mean?

5

This topic was automatically closed after 15 days. New replies are no longer allowed.