Setup Rate Limiting on multiple URL's

strong textWe have the following structure of URLs on our website. These URLs count runs in millions.

We want to rate limit these URLs. Something like 1000 calls every minute on each individual URL.

https://abc.website.com/hook/link1
https://abc.website.com/hook/link2
https://abc.website.com/hook/link3
https://abc.website.com/hook/link4
https://abc.website.com/hook/link5
https://abc.website.com/hook/link6
.
.
.
https://abc.website.com/hook/link9999999

We tried using Cloudflare and setup a rate-limiting rule with the condition mentioned:

'https://abc.website.com/hook/*

The problem is that it takes all the URLs at once and put the rate limit on overall URLs as a group and not on individual URLs. This means all our URL links as a whole group now get 1000 calls every minute and not each individual URL.

Can someone guide me on how to set up the rate limits on the URLs using Cloudflare?

This is expected behavior. You’ve put them all in one bucket, so Cloudflare keeps a count of how many times that bucket has been accessed by an IP address.

What you’re asking is for Cloudflare to allocate millions of buckets for your rate limiting, and that’s an expensive proposition. There’s a reason Cloudflare gives a set number of Rate Limiting rules per plan.

1 Like

So does Cloudflare offers the feature in the expensive Enterprise plan?

It’s the same feature, but Enterprise can have 100 rules (buckets).

I’m not sure why you need individual counts for individual URLs. That would allow the same IP address to hit all your endpoints at the same time, up to 1000 times per minute each. Ten endpoints means someone can hit 10,000x per minute. Your “millions” of URLs now become literally billions of requests per minute from a single IP address.

It would make sense to just not let a single IP address do this to your service.

The bottom line is that Rate Limiting won’t work in the way you describe.

1 Like

As pointed out above, the approach you describe doesn’t make that much sense; would you consider sharing your use case and why you need this behavior?

If you really want this, you can run the rate limit from the webserver directly or logs and then push bans to an IP list. Manage IP List items · Cloudflare Firewall Rules docs

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.