Cloudflare Firewall JavaScript detections blocking Google bot

Security
#1

Cloudflare JavaScript detections blocking Google bot.

I recently noticed that my website technobird.com got completely eliminated from Google search results.

I tried URL Inspection tool in Google Search Console, it was throwing a General HTTP error (without much details). Even though I was able to open the website in browser without any issue.

I contacted Google Search team about the issue and here’s the response:

Thanks for reaching out. While your site can be accessed via a browser, Googlebot has a different behavior. This could be due to a protection system that’s blocking unknown requests to your site.

To learn more about Googlebot and other Google crawler Verification, check this article. Please reach out to your hosting service provider to check what’s the underlying issue and fix it to avoid further access issues.

I also opened a ticket with Cloudflare in which I explicitly mentioned that the website is working fine when I open it in browser and the error only occurs when Google bot tries to crawl the website. So apparently its Cloudflare blocking requests from googlebot

And the response I received was a canned reply asking for irrelevant things which clearly indicates that the support rep didn’t even bother to read the information mentioned in the ticket.

Thank you for contacting Cloudflare Support. We’re sorry to read that you’re experiencing difficulties.

In order to better assist you with the problem you are experiencing, we will need some additional information from you.

Can you please share the following with us:

  • The specific error messages being returned and/or behaviours where you are seeing issues while on the website.
  • Specific step by step instructions on how to reproduce on our end - e.g. if this issue is only replicable behind a login, can you provide a temporary test account for us
  • A screenshot of the errors you are seeing.
  • Any relevant access logs from your web server.
  • A HAR file demonstrating the issue.

Please respond with that information as soon as you can so we can continue to work with you to resolve your issue.

Helpful resources

Kay Lee | Cloudflare Support Engineer

Can anyone please guide how to have it fixed?

#2

Which JavaScript detection is doing this?

Cloudflare doesn’t block legitimate Google bots by default, but if it’s being blocked, you should see the block in the Firewall Events Activity Log.

Another approach would be to try to “crawl” your site with a browser set to use the bot’s User Agent String:

#3

I think its Bot Fight Mode, specifically Javascript Detections, this domain is on Free plan and as per https://developers.cloudflare.com/bots/reference/javascript-detections I cant disable Javascript Detections for this domain.

I created a list of IP addresses in my account which includes IP addresses of Google Bot (https://developers.google.com/search/apis/ipranges/googlebot.json)

Then created a firewall rule (ip.src in $googlebot) to allow traffic from Google Bot. But it seems that Google bot is still not able to access my website. And as per Google its some protections system (Cloudflare probably) which is blocking Google bots.

#4

It looks like you’re not the only one:

#5

Any idea how to have it fixed?

I see Bingbot being blocked on my other website as well which is on a paid plan.

#6

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.