CF doesn't recognize Google IPs in Super Bot Fight Mode

Website, Application, Performance Security
1

What is the name of the domain?

example.com

What is the issue you’re encountering

Googleot not being allowed

What are the steps to reproduce the issue?

What a disaster. I had a feeling there were issues with my site and Googlebot.

I noticed in events CF is categorizing a legitimate Google IP as “definitely automated”. The IP can be found here as well https://developers.google.com/static/search/apis/ipranges/googlebot.json

And yes I have verified bots set to allowed.

Screenshot

Screen Shot 2024-11-08 at 1.01.38 PM
Screen Shot 2024-11-08 at 1.01.38 PM2082×1620 160 KB

2

Could you click on the particular Event below to show more details if it was GoogleOther as User-agent or something else under the Security → Bots https://dash.cloudflare.com/?to=/:account/:zone/security/bots? :thinking:

If you have got Block AI Bots enabled as well, might be triggered and blocked as shown on the screenshot.

Is verified, but is AI bot, therefrom by the enabled feature/setting, is blocked as expected.

slika
slika988×489 23.2 KB

3

Hi @fritex

I have Block AI bots disabled

Screen Shot 2024-11-09 at 1.46.56 AM
Screen Shot 2024-11-09 at 1.46.56 AM2052×940 74.9 KB

{
  "action": "managed_challenge",
  "clientASNDescription": "GOOGLE",
  "clientAsn": "15169",
  "clientCountryName": "US",
  "clientIP": "66.249.77.160",
  "clientRequestHTTPHost": "www.example.com",
  "clientRequestHTTPMethodName": "GET",
  "clientRequestHTTPProtocol": "HTTP/1.1",
  "clientRequestPath": "/",
  "clientRequestQuery": "",
  "datetime": "2024-11-08T10:49:23Z",
  "ref": "",
  "rayName": "8df4feffeb3ce811",
  "ruleId": "874a3e315c344b1281ad4f00046aab6f",
  "rulesetId": "48ba18287c544bd7bdbe842a294f1ae2",
  "source": "firewallManaged",
  "userAgent": "GoogleBot",
  "matchIndex": 0,
  "metadata": [
    {
      "key": "ruleset_version",
      "value": "7"
    },
    {
      "key": "version",
      "value": "6"
    },
    {
      "key": "type",
      "value": "customer"
    }
  ],
  "sampleInterval": 1
}
4

Thank you for feedback.

From the Ruleset, the Bot Fight Mode for Definite Bots → Managed Challenge.

May I ask what have you got configured for SBFM? :thinking:
How about Verified bots? :thinking:

slika
slika1000×1152 61 KB

Seems like legit one :thinking:

  • crawl-66-249-77-160.googlebot.com
1 Like
5

I have Verified bots Allow and Block AI Bots disabled.

6

Same here + when i try to add a custom rule in the WAF it doesn’t deploy…

1 Like
7

That doesn’t match the casing for Googlebot we’re familiar with nor what’s listed in Google’s documentation:

Are you able to see the corresponding indexing errors in the developer console ?

8

Good point but not sure we can hang our hat on that.

The IP cant be manipulated correct? The IP is listed with Google…

No errors but i also wouldnt look into that too much. I had some site speed issues and noticed Google visiting the homepage alot, this could be some sort of check that they dont let us know about in the console.

9

We are investigating with Google and will take action accordingly.

In the meantime you can use Custom Firewall rules to allow this traffic if you want or disregard unless you can confirm with Google that these blocks are causing some detriment to your website.

2 Likes
10

Glad to hear and just to update on the casing you mentioned earlier, if seems CF shows it as GoogleBot in the UI (Source browsers) but in the actual user agent its Googlebot …

For now I just have the option disabled and will monitor traffic to block curl agents and such. There was a recent Google core update so that could be the reason for the ups and downs I’m experiencing in SERP’s but cant take a chance having any Google traffic getting blocked.

11

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.