So if i have my website.com through standard CF setting. then can i turn on rate limiting on website.com/*? This is because each normal request to the website has around 60 assets (Css, images, js) to download. If i turn on rate limiting to say 50 requests per minute then it could potentially block legitimate traffic.
I would like to have negative rules to make sure that if cache is invalidated then Cloudflare does not treat normal requests as a rate limit attack.
It does, thanks for the clarification. I’ve taken the feedback for our product team. In the short term, I believe currently we’re allowing all plans to set longer sampling periods via the API which can allow for more focused rules and we are testing a “bypass” feature for rules with our Enterprise customers (so one could theoretically exclude example.com/*.css or example.com/static).
The challenge is that one could potentially mount a denial of service attack against a host by requesting assets which aren’t cachable or which had a short cache value set. So as we get more feedback from customers like you and real world usage statistics we’ll be providing more functionality and tweaks to make the feature more useful to customers.
IMHO rate limit just make sense for those requests that are not cached, as stated by @cs-cf. So, it doesn’t have to take into consideration requests for static assets like js, css, images, etc.
If the cache have a stale version of any static asset, Cloudflare proxy pull it from origin server at the next request and then cache it again. Nginx has a cache feature that put subsequent requests in a kind of queue when there is a resource being cached, so just the first request will reach origin.
I’m pretty sure that Cloudflare make good use of, or even better improved, this nginx feature.
I hope that could help you to clarify your question.
maybe allow Cloudflare users to set a custom header on their origin backends which they can setup in Cloudflare rate limiting to detectthat custom header on their origin backends and exclude or include them in rate limiting at Cloudflare level ?
Not a bad idea. We are currently testing a bypass feature with our ENT customers which allows you to exclude paths/patterns, but the option for origin to educate us about what resources are OK to be pulled at any rate would be interesting.
Since rate limiting is such a new feature I anticipate we’ll see a number of iterations to tweak and improve the feature over time. We have some other cool items on the roadmap this year which I think will also also help with the larger problem trying to be solved.
From a personal standpoint my worry/concern is that we roll out these kind of features in a way that a non-expert user can take advantage of them without having to be an expert (while still allowing power users to tweak the settings to get even more out of them).
I was actually a beta tester of the Cloudflare Rate Limiting Product, it’s nice in that Cloudflare does the rate limits at their edge rather than the user’s origin server. Because of this the already overwhelmed origin doesn’t receive additional tasks and become unable to keep up with the tasks. It’s very well implemented.
I guess I am a bit late to this party, but I am looking at enabling rate limiting as well and have some of the same questions. I just want to make sure that my assumptions are correct (as I’m a bit of a noob at this).
I have a page that makes 94 requests upon page load (it’s a list that contains quite a lot of images). Of those requests 68 are images, 9 are js, 5 are css, and 4 are fonts. The remaining 8 are XHR / Other requests.
Now, anything that is cached within Cloudflare will not count as a request if it is successfully retrieved from said cache. Would this statement be true?
So, in theory, since those 68 images are static, they should be cached in Cloudflare. So those shouldn’t count. Same would go for the JS / CSS / Fonts I would assume (which some of those are served from a cdn anyway).
So, in theory, this page loading would potentially hit the rate limiter would tally 8 requests in the two to four seconds it takes this page to load.
Am I following along the correct path, or am I completely helpless? Somewhere in between? Haha. Thanks in advance.
Yes, in theory. In practice it depends on are they cached. Maybe the cache will be flushed and you will have a new request again.
A new request from different geo position will hit different CF DC, on which if it is not cached already (and if not using Argo) will hit the origin again.
You will have to set the limits yourself, as you can see what rates are OK for you.
Take a look at the bot rates, at the visitor rates at the site/hosting acceptable rate.
Keep in mind that in one second a bot and a human, can make the same number of requests.
In 1,5 or 10 minutes, not so much.