My website is blocked to Google crawl

Hi, this is Amir. My website is blocked to Google Crawl, someone else set my Cloudflare account, and we don’t understand why it is blocked for Google to crawl.

Help, please.

I also tried to add some rules to the firewall security, and nothing is happening.

I don’t see any firewall rules that would block, if you are still having an issue, check robots.txt perhaps.

Hi Cloonan, thanks a lot for your time. This is the Robot.txt I have on my site.

The firewall rule is probably the cause, and in reality it will not be very effective, I can bypass by pretending to be Google!

 % curl  --dump-header - --silent -o /dev/null | grep HTTP
HTTP/2 403 
 % curl -A Googlebot --dump-header - --silent -o /dev/null | grep HTTP
HTTP/2 200

If you really want to block users outside the US, a better rule might be to just allow Known Bots:

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.