Google Bot cannot crawl the page

Hi,

I was trying to make a live test of a page in Google webmaster tools but cloudflare is so damn complicated and absolutely no longer userfriendly since all the changes. It is impossible for me to find out why the google bots are blocked all the time.

I was putting the IPs from the google bots into the WAF Rule but all the time I get the message in google webmaster tools

“Page cannot be indexed: Blocked due to access prohibited (403).”

@ CLOUDFLARE TEAM you have to do something against this ■■■■. Really. It cannot be so difficulty for you to filter out the google ips so that your clients don´t have the trouble each time!
I have no idea what else I can do… ! Before the changes on your cloudflare page I was able to fix it by myself…
It makes me angry.

Hi!

Sorry you are having trouble with this.

My advice would depend on the zone settings, as Cloudflare might be blocking Verified bots such as Googlebot based on your settings or a rule you have set up.

A good start would be to filter out Security events by Google’s ASN, and check for events that match Googlebot’s IP Ranges and User Agents and what is blocking them and adjust settings, IP Access Rules / create Custom WAF rules or Managed Rules exceptions.

Hope it helps.

Greets

Thanks

1 Like

Hi sgoncalves,

May I know how to do the setting? Kindly advise me step by step, thanks!

Rick

@rick40 when you comment on a thread from someone else, you’re less likely to get a reply that is helpful for you. On this thread, let us know what you’d like to accomplish. I suspect it’s a rules blocking the bots. You’d find detail under security → events and create the rule under security → WAF.

Set up Firewall Rules: Go to the Security → WAF app in your Cloudflare dashboard and create rules to challenge or block traffic based on characteristics such as country, user agent, or IP address.
Screenshot 2024-02-02 at 4.56.36 PM

Go here, https://dash.cloudflare.com/?to=/:account/:zone/security/waf/custom-rules

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.