Overactive Rate Limiting

I went months without having users complain about rate limiting until a few weeks ago. It was set at 20 requests in one minute. Now I get a complaint every few days. I have relaxed it to 45 requests in 10 seconds, and people are still being rate limited.

Today, a user told me he was particularly careful to wait for 10 seconds before attempting to load another page, and he STILL was rate limited.

I guess I don’t understand the concept, but I can’t afford to turn it off. It’s the only thing keeping me from having to shut the site down due to bots that constantly index the site. If I turn off rate limiting, I’ll be charged overages by my webhost. If I leave it on, it constantly annoys my legitimate readers.

What setting would you use to rate limit a site that has many pages with just text and a couple of images, but a few pages that might have as many as 50 or 60 images?

And are there any theories on why this seems to have suddenly become an issue when it hasn’t been an issue for more than two years using rate limiting?

I don’t have any experience with Cloudflare rate-limiting however, you can modify the Firewall in the Cloudflare dashboard to help stop bots.

If you go to Firewall -> Tools you can enable the Bot Fight Mode which as it describes will: Challenge and/or block requests matching patterns of known bots before they can access your site.

alongside that, you can create a rule to check if the Threat Score is above a threshold and block or challenge the user. Here is an example rule:

For more info on firewall rules, you can look here: https://developers.cloudflare.com/firewall/cf-firewall-rules

Thanks. I’ll definitely look into those options.

In general, rate limiting of images is not particularly useful, and Cloudflare does not apply rate limiting rules to cached content. You are normally attempting to prevent either abuse of an application or overloading of your origin server, database etc.

The first thing I would do is verify that all content that should be cached is cached, and that that the max-age is set appropiately.

I have Caching Level set to “standard” which is what Cloudflare recommends. Would “no query string” or “ignore query string” be better to ensure images are cached?

Or would links to images be considered “HTML static content” requiring a separate page rule to be written?

Under Browser Cache TTL, I had that set to 1 month, so I just changed it to the max allowed, which is 1 year.

My images are all .png format and they’re in several sub-folders, so I wrote this page rule and set it to “cache everything.”

Am I on the right track?

*.MyDomain.com / * / *.png

(there are no spaces…I had to add spaces to get the rule to display every character here.)

Thank you to everyone who has made suggestions. I appreciate the help.

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.