Google - sitemap could not be read HTTP Error: 403

[Roberta]

I just purchased a domain from Cloudflare and started creating a WordPress website just to see that google search console doesn’t fetch my sitemap. I’ve read and tried every solution offered in the other posts regarding this matter but it’s still not working. When I pause Cloudflare, everything works. So what should I do?

[fritex]

@Roberta

When I pause Cloudflare, everything works. So what should I do?

You should see the challenged/blocked firewall events in the firewall events if you navigate to the Cloudflare dashboard → Security → Overview and lookup for Firewall events for the past 24hours or so. Once you find them, click on a particular one to find more details about it (user-agent, IP, HTTP version …). If yes, could you share some details which service was triggered that blocked you? Could be Browser Integrity Check or some other

• you should see your origin host/server IP out there and user-agent, etc.

It knows to happen due to the WordPress using HTTP/1.0 and empty user-agent, therefore while executing WP-Cron or some other related JSON/REST API request via plugin which triggers the WAF rules (as it should normally).

[Roberta]

Hello and thanks for your reply!

There are 8 events over there, all from yesterday:

The only Browser integrity check is this:

What should I do next?

[fritex]

What happens if you temporary disable Browser Integrity Check feature?

Googlebot is on the “good” and “known” list:

Firewall Rules FAQ · Cloudflare Firewall Rules docs

Google is also on a known list of “good bots”, despite there are again, the “bad” requests from the Google AS number from which Googlebot has got official IPs published too:

• https://developers.google.com/search/apis/ipranges/googlebot.json

Nevertheless, you could try to create a Firewall Rule to allow/bypass it, if that somehow happens.

What I use to allow only Googlebot to access the sitemap.xml and robots.txt, sharing my below post which contains an Firewall Expression and a screenshot of allowing only Googlebot + Bingbot to robots and sitemap using a Firewall Rule:

FW rule blocking Microsoft ASN without blocking Bing

You might want to switch the toggle button from green to gray. Therefore, make sure the method is “block”. However, I might not be sure how Cloudflare checks this, by user-agent or IP range So, maybe like writing your own Firewall Rule … for example: (http.request.uri.path contains "sitemap" and ip.geoip.asnum ne 15169 and not http.user_agent contains "Googlebot") or (http.request.uri.path contains "sitemap" and ip.geoip.asnum ne 8075 and not http.user_agent contains "Bingbot") or…

Helpful article to find out more about this feature:

Understanding the Cloudflare Browser Integrity Check · Cloudflare Support docs

Nevertheless, regarding 403 error:

Community Tip - Fixing Error 403 Forbidden

Error Try the suggestions in this Community Tip to help you fix Error 403 Forbidden. Background A 403 Forbidden Error is a client side error that means that the client sent something the origin was unable to process. With the exception of requests that violate WAF rules or subdomains that are not covered by a certificate, Cloudflare does not generate any 4xx errors, so this would indicate something is not configured correctly with your hosting provider or your client is sending something inco…

Possible solutions:

1. Creating a Firewall Rule with cf.client.bot “true” and action “allow”, make sure it is the 1st Firewall Rule from above.
2. Creating a Firewall Rule with user-agent contains bing and action “allow”, make sure it is the 1st Firewall Rule from above.
3. Risky one but … add Google AS number (AS15169) to “allow” in IP Access Rules.

[Roberta]

I disabled Browser Integrity Check but nothing happened.

I created 3 rules as you told me:

1. (cf.client.bot) Allow
2. (http.user_agent contains “bing action”) Allow
3. (ip.geoip.asnum eq 15169) Allow

Still, nothing happened but for the first rule, I got 4 new events, every time I tried to add the sitemap to the google search console. It looks like this but still cannot fetch the sitemap:

[Paige]

Hello, I have requested more troubleshooting information in the ticket you currently have open.

[Roberta]

Hello, I replied to you in the ticket.

Here is the reply:

I don’t understand. For what webpage do you need this thing? My problem is with sitemap.xml
This sitemap: XML Sitemap - Mental Shortcuts mentalshortcuts. com/sitemap.xml
It cannot be added to Google Search Console