Why is googlebot blocked

[koxmarcel]

What is the name of the domain?

erosweb.com

What is the error number?

Googlebot is blocked

What is the error message?

403 error

What is the issue you’re encountering

that Googlebot and googlemobilebot and googleadsbot is being blocked

What steps have you taken to resolve the issue?

1. disabled bot blocking and javascript to find bots earlier
2. added a waf rule to allow everything from the asn address of google

Still when i do

curl -I -A “GoogleBot” https://www.erosweb.com

I get this below in ssh.

HTTP/2 403
date: Thu, 28 Nov 2024 10:27:11 GMT
content-type: text/html; charset=UTF-8
content-length: 4517
x-frame-options: SAMEORIGIN
referrer-policy: same-origin
cache-control: max-age=15
expires: Thu, 28 Nov 2024 10:27:26 GMT
report-to: {“endpoints”:[{“url”:“https://a.nel.cloudflare.com/report/v4?s=Rp KazCF8Z6w1hLtDI2e%2BJWg6nSJKIyZxkztgoZibYBVkbc4iITbBLvZdqwTpnJwxMlG3loeisFJ6A7%2 FCLFujKqz3WOtAZvNBNh4cli3XbftZNZeOAOgM2ippLSmlXENsjfeMZA%3D%3D”}],“group”:“cf-ne l”,“max_age”:604800}
nel: {“success_fraction”:0,“report_to”:“cf-nel”,“max_age”:604800}
server: cloudflare
cf-ray: 8e99a9fae8810b07-AMS
server-timing: cfL4;desc=“?proto=TCP&rtt=2420&min_rtt=2315&rtt_var=725&sent=5&re cv=9&lost=0&retrans=0&sent_bytes=3395&recv_bytes=827&delivery_rate=1210196&cwnd= 252&unsent_bytes=0&cid=77be82c1d92fd786&ts=17&x=0”

[micky]

Hey there,

Looks like the domain you shared with us is no longer active in Cloudflare, so it is difficult to say with certainty what issue you were having with Googlebot.

Based on the information you have shared with us, the curl command you did was blocked as this was emulating GoogleBot, but was not coming from one of Googlebots IPs, so there’s a good chance this would have been identified as a non-verified bot instead of as a verified bot, leading to the 403 status you got. If you choose to use Cloudflare again later for this domain, and you keep encountering this issue, we could further look into this.