I am using the Rate Limiting service for a while now. Today I have noticed there have been a lot of Googlebot requests blocked by the Rate Limiting service. I had to turn of the service immediately. I thought Cloudflare whitelists Googlebot IP’s by default. Is it not the case?
Btw I have checked the log again while I’m writing this post, and even though I have turned off Rate Limiting a couple of minutes ago, I can see Googlebot requests are still being blocked for “Rate Limiting”, should I turn off Cloudflare completely? This isn’t supposed to happen, right?
Thanks for any input