403 Error With Google Bots

Security
#1

Hi,

So it looks like Cloudflare is blocking Google Bots on my website. Using https://httpstatus.io/ I can check if Google Bots can crawl my site, but it’s returning a 403 error on all pages, but specifically: https://noisypixel.net/sitemap_index.xml

Screen Shot 2021-02-19 at 6.39.58 PM
Screen Shot 2021-02-19 at 6.39.58 PM2098×1236 201 KB

Let me know if this is something that I have accidentally set on my side or what I can do to fix this. Thanks for the help!

#2

Here’s another screenshot of the issue showing that I’m using the User Agent Googlebot.

Markup 2021-02-20 at 09.29.24
Markup 2021-02-20 at 09.29.241481×755 27.3 KB

#3

Take a look at your Firewall Events Log and that should show you the security setting that’s blocking those requests.

#4

Ooooh, I see, it’s catching it as a fake Googlebot:

Screen Shot 2021-02-19 at 7.05.54 PM
Screen Shot 2021-02-19 at 7.05.54 PM2022×910 109 KB

Do you think that this is only the case for this tool and not Googlebot proper?

#5

A Google bot that runs on AWS? That sure doesn’t look right.