Googlebots are being served 403 headers by Cloudflare

Website, Application, Performance
1

My sitemap and other pages are showing as not indexable and/or redirect errors on Google Search Console and when I run a check via Redirect Checker | Check your Statuscode 301 vs 302, the standard version is code 200, but switching to Googlebot is returning 403 errors.

I already checked with my host who wrote:

**I noticed that when I choose Google or Bingbot from the Set User-Agent drop-down, the response indeed is 403, however, no logs are being recorded on our end - as if the request does not reach the SiteGround server at all. After that, I noticed that the headers are actually returned from Cloudflare.**emphasized text

Any suggestions of what I should check next?

4

Go to Security… Events… in the Cloudflare dashboard and you should be able to find the reason for the rejection if it is being blocked at Cloudflare.

Note that using that tester and pretending to be the Googlebot may trigger a block by Cloudflare’s bot tools since it won’t be coming from the network/IP range expected (anyone can set a user agent string to anything). You need to find the events relating to the real Googlebot…

1 Like
5

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.