What is the name of the domain?
What is the issue you’re encountering
I’m currently on the Free Plan with Cloudflare, and I found that I can apply rate limiting to Search Engine Crawlers in general. For example, I could limit requests to 3 per 10 seconds and block them as the action. However, what I ideally want is to rate limit Googlebot to 20 requests per minute and apply a Challenge (instead of blocking) when it exceeds the limit. This would help to reduce server load while still allowing Googlebot to crawl my pages. My question: Is it possible to set this kind of rate limiting with a Challenge action for Googlebot specifically on the Pro plan (or higher)? Or is the rate limiting feature on the Free plan too basic to apply custom actions like a Challenge, and would upgrading to the Pro plan give me more control over this? I would appreciate any insight from those who have used this feature or encountered similar challenges. Thanks!
What is the current SSL/TLS setting?
Full
