[Support Tickets Ignored] Cloudflare is blocking Googlebot since 15 Feb (Confirmed)

[user5203]

What is the name of the domain?

mybro.tv

What is the error number?

429

What is the error message?

Too Many Requests

What is the issue you’re encountering

Cloudflare ignoring custom rules

What steps have you taken to resolve the issue?

1. I created custom rule security/waf/custom-rules, described in details allowlist for google, to suppress all restrictions:

• All remaining custom rules
• All rate limiting rules
• All managed rules
• All Super Bot Fight Mode Rules
• Zone Lockdown
• User Agent Blocking
• Browser Integrity Check
• Hotlink Protection
• Security Level
• Rate limiting rules (Previous version)
• Managed rules (Previous version)

2. I contacted support 2 times, no response

• The issue with real googlebots still exists, everyday cloudflare is blocking real google crawlers
• The crawlers are not new, they are 20-30 days old
• Cloudflare internal rules have priority over my custom rules

Was the site working with SSL prior to adding it to Cloudflare?

No

What is the current SSL/TLS setting?

Full (strict)

What are the steps to reproduce the issue?

1. Go to security analytics dashboard and filter out requets by asn and status code analytics?status-code=429&asn=15169

2. Get source ip, like 66.249.68.2

3. Resolve ip, perform dns lookup
   host 66.249.68.4
   4.68.249.66.in-addr.arpa domain name pointer crawl-66-249-68-4.googlebot.com.

4. Confirm that this is real crawler
   Googlebot and Other Google Crawler Verification | Google Search Central  |  Documentation  |  Google for Developers

Screenshot of the error

[Laudian]

Please go to the events page to find out what triggered the 429:

https://dash.cloudflare.com/?to=/:account/:zone/security/events (select your domain).

Please also share the expression of your allow rule.

[user5203]

(cf.verified_bot_category in {“Search Engine Crawler” “Search Engine Optimization” “Monitoring & Analytics” “Advertising & Marketing” “Page Preview” “Academic Research” “Security” “Accessibility” “Webhooks” “Feed Fetcher” “AI Crawler” “Aggregator” “AI Assistant” “AI Search” “Archiver” “Other”}) or (cf.client.bot) or (http.user_agent contains “Google”) or (http.user_agent contains “Bing”) or (http.user_agent contains “bot”)

I can confirm that I have 174.04K out of 174.09K requests properly skipped security check. But I am still confused because in google search console I see that we have 14% failed requests to our domain and this incident is on the edge because we don’t have any logs on our server related to this situation

[Laudian]

I’ve asked someone to take a look at this.

[0x03]

same question, got 429 since feb 15

[user5203]

for those who is still have questions disallow cdn-cgi