Googlebot is excluded from traffic filtering, Browser integrity Check is OFF, but Google Search Console and httpstatus test still marks pages with errors
What steps have you taken to resolve the issue?
I have reviewed all my Cloudflare settings and asked my Host Provider to do so as well. We haven’t found unusual configuration. Everything appears to be in order. As soon as Cloudflare is paused, the errors are gone.
I also having this issue with Google and bing. When CF is paused there is no issue, no note in any logs and I have allowlisted everything for sitemaps still doesn’t work have Cloudflare is enabled and get 5xx
I’ve thoroughly investigated this issue with the senior technicians at my hosting provider. They found nothing unusual on their end, and they also helped me review all Cloudflare settings: everything appears to be in proper order.
Despite having Google is excluded and the skip operations are visible in the Events log, both google and httpstatus tests fail with Cloudflare.
Since these errors disappear immediately when Cloudflare is paused, it strongly suggests the issue is originating from Cloudflare’s side. I’m genuinely disappointed that no staff - one from their support team has acknowledged this yet.
I had a step in the right direction which worked for me, not ideal, ssl settings I reduce from full strict to full therefore not enforcing the CF cert at origin. Seems to be working for now.
If you’re getting a 502 or 504, can you attach a screenshot of the page?
The 502/504 errors are caused by a problem connecting to an upstream server - meaning your server is trying to initiate a process and this fails to work as expected or times out. In most cases of 502 / 504 errors, back-end servers are not communicating correctly. When this happens, you will see color page with Cloudflare branding and the Error 502Bad Gateway or Error 504 Gateway Timeout. Review this Community Tip for fixing 502 or 504 gateway errors.
I thought this might provide valuable insight. To summarize: the pages load correctly, but it appears that Cloudflare is blocking something, and it seems to be an issue outside of the user’s control, one that Cloudflare staff is not aware of yet.
As soon as I pause Cloudflare, the issues are gone.
I understand what you mean by the color page, but in this case the affected pages load correctly, without the cloudflare error/splash page you mentioned, still fail at Google and httpstatus tests with Cloudflare..
As per your request, I attach the screenshot that the page loads perfectly
Thank you for your tip - it didn’t help in my situation. I tried Full/Flexible/Off settings and Google still can’t crawly my site because of 5xx errors. The pages load perfectly on my browsers, the Search Console error is gone only when cloudflare is paused. Isn’t there something else as well you have tweaked around that time?
I’ve gone through several rounds of troubleshooting on my end and have confirmed that the issue is indeed related to Cloudflare. Here’s a summary of my findings so far:
Firewall Rules: I’ve disabled all firewall rules, and Googlebot is properly excluded from any traffic filtering. No custom rules are blocking its access.
Cache Clearing: I’ve cleared the cache multiple times, but the issue persists.
SSL/TLS Settings: I’ve tested various SSL settings (Full, Flexible, and Off), but this doesn’t resolve the issue.
Cloudflare Features: I’ve also turned off Bot Fight Mode and Rate Limiting, and tested with Rocket Loader off (as recommended).
Behavior with Cloudflare Paused: The errors disappear immediately when I pause Cloudflare, indicating that Cloudflare is the source of the issue.
Googlebot Logs: My server logs show that Googlebot is not being blocked, and the traffic is being passed through Cloudflare correctly.
I’ve also noticed that pages load fine in browsers, but Google Search Console and HTTP status tests continue to report 502/504 errors when Cloudflare is enabled.
Could you offer any insights into further steps I can take, or if this is an issue specifically with Cloudflare’s infrastructure or configuration? I’d greatly appreciate any help in resolving this, as it’s impacting SEO and crawlability.
The issue was resolved and mainly was related to overly large or duplicated cookies (webp_support) causing the browser and possibly Cloudflare’s WAF to reject or mishandle requests.
Multiple webp_support=0 cookies were being set repeatedly, pushing the total cookie size beyond the 8190-byte limit.
This led to HTTP errors or blocks, making Googlebot and other clients unable to properly access the site.
Cloudflare’s WAF (or security rules) was likely blocking requests due to suspicious cookie sizes or patterns.
