Exclude GoogleBot / Bing from WAF rate limiting bot rule?

Website, Application, Performance Security Center
1

I’m using Security > WAF > rate limiting rule to prevent bots from overloading my site. I use the “verified bot” selector. However, I find that I have to set the rate higher than I’d otherwise like to because I don’t want to limit GoogleBot and Bing. I see that GoogleBot is getting blocked with my current rule.

According to this Cloudflare rate limiting Google bot, the GoogleBot is excluded from rate limits, but that’s not what I’m seeing. I’ve verified that the blocked IP address is Google’s.

Is there a way to allow-list certain bots? Thanks!

4

If you set it to exclude Verified Bots, this shouldn’t be happening. Could you please share the expression of your Rate Limiting rule, as well as a recent WAF Event where a search engine was blocked?

5

This is the expression:

(cf.bot_management.verified_bot and not http.request.uri.path contains “/log_pageload/” and not http.request.uri.path contains “/media/”)

And this is one of the blocked requests:

Ray ID
80195acd9b482fd4
IP address
66.249.73.10
ASN
AS15169 GOOGLE
Country
United States
User agent
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
HTTP Version
HTTP/1.1

6

As it stands, the rule is only rate-limiting requests from verified bots, the opposite of what you want.

The verified bot field will return true when the request comes from a verified bot. You must negate it as part of the expression so that it does not rate limit verified bots.

(not cf.bot_management.verified_bot and not...)
7

I guess I assumed that “verified bot” meant all things that Cloudflare has identified (verified) as a bot, but that you excluded Google bot from this set.

What I want to do is to exclude non Google/Bing bots that sometimes come on too strong. These bots usually identify themselves as bots in the user agent. I don’t want to risk rate limiting my actual users. Actual users can sometimes have a high rate because of Ajax calls and checks and such. Is there any way to do this?

Is “verified bot” everything that Cloudflare has identified as a bot, including Googlebot as well as bad(ish) bots? And “NOT verified bot” is everything else, including my users?

I see now that switching off the green “value” button makes it a NOT verified bot, which is perhaps not the most obvious UI…

(Sorry I didn’t respond sooner. Was on vacation.)

8

No. Cloudflare Known Bots represent bots such as from search engines, social networks etc., that can be verified as being the legitimate bot they claim to be (many hackers try to impersonate Googlebot, Bing etc. by copying their user-agent string.)

If your intention is to block visitors, you should toggle it off, and the expression will look like

not cf.client.bot and ... (the rest of your conditions)

To block bad bots Cloudflare offer its bot products, such as Bot Fight Mode, Super Bot Fight Mode and Bot Management.

9

Thanks for your reply and the suggestions, I will look into those other services. I guess I don’t quite see the purpose of the rate limiting tool. The two options are to either risk limiting my actual users, which I do not want to do, or to limit bots, including good bots, which I do not want to do. I suppose I could set a very high limit, just to limit the absolute worst offenders, then decide whether that should be the bots or non bots.