Google - sitemap could not be read HTTP Error: 403

[fritex]

What happens if you temporary disable Browser Integrity Check feature?

Googlebot is on the “good” and “known” list:

FAQ · Cloudflare Web Application Firewall (WAF) docs

Google is also on a known list of “good bots”, despite there are again, the “bad” requests from the Google AS number from which Googlebot has got official IPs published too:

• https://developers.google.com/search/apis/ipranges/googlebot.json

Nevertheless, you could try to create a Firewall Rule to allow/bypass it, if that somehow happens.

What I use to allow only Googlebot to access the sitemap.xml and robots.txt, sharing my below post which contains an Firewall Expression and a screenshot of allowing only Googlebot + Bingbot to robots and sitemap using a Firewall Rule:

FW rule blocking Microsoft ASN without blocking Bing

You might want to switch the toggle button from green to gray. Therefore, make sure the method is “block”. However, I might not be sure how Cloudflare checks this, by user-agent or IP range So, maybe like writing your own Firewall Rule … for example: (http.request.uri.path contains "sitemap" and ip.geoip.asnum ne 15169 and not http.user_agent contains "Googlebot") or (http.request.uri.path contains "sitemap" and ip.geoip.asnum ne 8075 and not http.user_agent contains "Bingbot") or…

Helpful article to find out more about this feature:

Browser Integrity Check · Cloudflare Web Application Firewall (WAF) docs

Nevertheless, regarding 403 error:

Community Tip - Fixing Error 403 Forbidden

Error Try the suggestions in this Community Tip to help you fix Error 403 Forbidden. Background A 403 Forbidden Error is a client side error that means that the client sent something the origin was unable to process. With the exception of requests that violate WAF rules or subdomains that are not covered by a certificate, Cloudflare does not generate any 4xx errors, so this would indicate something is not configured correctly with your hosting provider or your client is sending something inco…

Possible solutions:

1. Creating a Firewall Rule with cf.client.bot “true” and action “allow”, make sure it is the 1st Firewall Rule from above.
2. Creating a Firewall Rule with user-agent contains bing and action “allow”, make sure it is the 1st Firewall Rule from above.
3. Risky one but … add Google AS number (AS15169) to “allow” in IP Access Rules.