500,000 requests by one IP address over 18-hr period. How can we prevent this?

Recently an IP address in Slovakia used some sort of script to make over 500,000 requests of various pages on our site over an 18 hour period. It wasn’t content-scraping or anything logical like that. Seems like it was just trying overwhelm our site. If I’m doing the math right, that’s an average of 7 requests per second.

So we blocked the IP address when we realized it had happened. But we’re trying to figure out what rule to put in place to prevent this kind of thing in the future. There’s no legitimate case where we’d want to allow that volume of requests.

However, I’m afraid to dabble in site-wide rate-limiting because I don’t want to block any legitimate search engine bots or anything like that.

Any tips for a general “don’t allow any one IP to overwhelm my site but allow the good guys” rules approach?

Another thing I’d also love to know is whether cloudflare can alert us when a huge unusual spike is happening so we can immediately look into it?

Thanks!

Rate Limiting doesn’t apply to legitimate search bots.

Weird that Cloudflare can’t apply heuristics in such cases. I have been sporadically getting targeted by single IPs in my Workers endpoint and Cloudflare just does nothing about this high RPS situation. Makes no harm, though, other than depleting my Workers quota.

Thanks @sdayman! I was wondering about that and I googled a bunch to see if that was true but wasn’t able to find that info. But now I do see it spelled out here:

Cached resources and known Search Engine crawlers are exempted from a customer’s Rate Limiting rules. Rate Limiting does not negatively affect a website’s SEO ranking.

1 Like

Regarding my question about cloudflare alerts, I do now see that this feature exists and you can get ddos alerts. I wonder if this would have qualified as a ddos alert though since cloudflare seemingly didn’t do anything about this at the time? There are no firewall incidents recorded for this IP address.

This topic was automatically closed after 30 days. New replies are no longer allowed.