Cloudflare returns 403 Forbidden to Googlebot

I don’t see how that is relevant; I didn’t ever say they are equal.

Unlike your example Security/Events report filtering the Source IP(s), mine just shows Action: Skip.

I have no clue then why Analytics shows “403” while Security shows SKIP. Neither, how and why you’ve got such reports in Google Search Console because of it :thinking:

Might be a question for Cloudflare support if these numbers are not in some co-relation and dispersed among two products from them.

Can you share the full Google User-agent(s) that are impacted ?

Can you also confirm that the UA from analytics matches events ?

2 Likes

In the last 24 hours, there were 110 instances of Googlebot being blocked with status 403 Forbidden on my web site (according to the Security/Analytics report page).

Source user agents are typical; nothing out of the ordinary:

  1. Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.6723.116 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

  2. Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.6723.69 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

  3. Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

  4. Mediapartners-Google

If I take the last record from the Security/Analytics report (Nov 12, 2024 6:03:04 PM)

The same/matching record from the Security/Events page:

Let me know if you need any other information.

Thank you for looking into this for me.

We tracked these down to be DDoS mitigations on your zone. Google is not immune from protection because the crawl rate got really high. Likey something related to the way URLs are formatted on your site, but Google would be more able to confirm specifics about their crawler behavior.

You can adjust your zone’s DDoS thresholds in the Dash UI. HTTP DDoS Attack Protection managed ruleset

Logs are confusing because of the overlapping managed challenge from DDoS returning the 403. Long term we hope to fix this to be more clear, but no ETA do to the complicated nature and broader impact.

1 Like

Thank you for all the work and feedback on this. I really appreciate it.

I understand. I am just relieved that it’s known and it will be looked into eventually.

Thank you again, Cloudflare Team.

1 Like

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.