Does Cloudflare tie "known bots" to IP address, and what if they change?

[gareth6]

I’ve been using the “known bots” rule to make sure that we don’t inadvertently block known, friendly bots (in particular, Google).

I wondered, does Cloudflare tie known bots to their respective known IP address ranges? I’m guessing they must, otherwise an attacker could simply spoof their user-agent as “GoogleBot” or “Yandex”, etc and get around firewall rules easily.

But in addition to that, what if a Bot’s IP address range changes (or they introduce new IP ranges)? I just wondered how it was possible for the folks at Cloudflare to stay on top of this in a reliable way?

I guess what I’m worried about is waking up one morning to find our websites all vanished from Google because it’s been inadvertently blocked.

[michael]

@gareth6

does Cloudflare tie known bots to their respective known IP address ranges?

In general yes.

All well behaved Bots publish a mechanism to validate their IP addresses. In the case of Google it is published here: https://support.google.com/webmasters/answer/80553?hl=en, Bing and Yandex use the same technique.

I don’t know how CF manage this internally, but I would expect this is not done in real-time, and that they also have a relationship with the main bots to flag any changes.