Security Bots,google spider seems to be blocked, how do I unblock it

I have added the indicates a ‘known good’ bot. rule
Security Bots,google spider seems to be blocked, how do I unblock it
The Cloudflare automatic firewall seems to block legitimate spiders like google, bing, etc. How do I unblock these IPs?

Those aren’t crawlers:

 % host 66.249.92.62      
62.92.249.66.in-addr.arpa domain name pointer rate-limited-proxy-66-249-92-62.google.com.

If it was a crawler, it would resolve as “googlebot”

1 Like
  • Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.41 Mobile Safari/537.36 (compatible; Google-AMPHTML)

That user agent string would say GoogleBot. User Agent String is easy to set at the browser level:

If you hover any of those entries, you should be able to click Filter, then scroll down to see which setting is blocking that request. Still, those aren’t legitimate crawlers.

When you say “Security Bots”, which ones are you thinking of?

Thanks, I was just wondering, what should I do if I come across a User Agent that I want to free.

For example, I want to put the following User Agent that.
Let him have normal access.
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.41 Mobile Safari/537.36 (compatible; Google-AMPHTML)

Going by User Agent isn’t the safest approach, as I already showed that those can be faked. Any legitimate service should have a list of IP addresses they use so you can free those instead.

You can add a Firewall Rule to Allow. I usually use a “Contains” statement and enter a unique portion of the User Agent String.

Still, it’s best to first figure out what setting is blocking those requests. Cloudflare blocks very little by default.

1 Like

But I seem to find that Security Bots ,have a higher priority level than WAF.

Yes, if you are using Super Bot Fight Mode, then you can not bypass that.

Did you find out which setting is blocking those requests?

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.