Cloudflare block GoogleBot and how to enable it?

Hi all,

I am using Semrush for SEO purpose and need to enable GoogleBot to crawl my website but Cloudflare still blocking it no matter how I allowlist or enable or even stop the firewall, how to fix it please?

Rick

Hi there,

Cloudflare should not block Google Bot unless you have some Custom Rules defined to explicitly block it - if you do have rules that are blocking it (eg. Blocking the United States) you can create an exception in your rule by using the cf.client.bot field. There is an example in this developer doc:

Hope this helps!

Hi Damian,

Thanks for your reply and I did the setting, but Semrush still can’t crawl my website (www.losthk.com). I attached some screenshots for your reference, please let me know if you can advise me any further solution.

Semrush allowlisting IP: Whitelisting SplitSignal Bot

Thank you
Rick

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.