I have to disable Cloudflare Specials for Googlebot to be able to index my site otherwise it gets 403 error from Cloudflare. I don’t have any firewall rules.
What’s the solution?
For starters, are you sure this really is Google? Cloudflare’s default settings won’t block Google.
Can you post a screenshot of the firewall log entry where Google got allegedly blocked?
2 Likes
Google Search Console is telling me that it tried to crawl my site as “Googlebot desktop” and encountered with “5xx” error which is a server error.
The problem goes away as soon as I disable “Cloudflare Specials”.
I got that solution from here
1 Like
2 Likes
Where is firewall log?
Cloudflare seems to do the right job here. 34.83.190.242 does not appear to be an address of Google’s search engine.
nslookup 34.83.190.242
Server: one.one.one.one
Address: 1.1.1.1
Name: 242.190.83.34.bc.googleusercontent.com
Address: 34.83.190.242
2 Likes
That’s probably not the correct firewall entry. If the legitimate Google Search Console is reporting a 5xx error, I wouldn’t expect GSC to be crawling from a cloud user VPS.
2 Likes
Plus, the request got blocked with a 400, not a 500.
1 Like
I enabled CF Specials and:
- disabled both #100201 and #100201_2: GSC ok
- disabled #100201_2 only: GSC not ok
- disabled #100201 only: GSC ok
So the culprit is #100201.
Disabling #100201 solved the issue.
You can certainly disable that rule, but you do need to realise this will allow anyone to pretend to be Google (such as in the screenshot you posted).
If that’s fine with you, then you are good to go.
I don’t want to disable any rule. I certainly want CF to investigate this issue and solve it. I just don’t know how to escalate this.
I hope they read this discussion.
Not sure what Cloudflare should investigate here. There is no apparent issue on Cloudflare’s side. They did exactly what you told them to do. Block requests which pretend to be Google but are not Google. The screenshot you posted is not from Google’s search engine nor did it send a 500.
That’s either the wrong screenshot - in which case my question from yesterday still stands - or Cloudflare did not block anything that it shouldn’t have.
2 Likes
With that rule enabled → GSC error
With that rule disabled → No GSC error
I trust Google on this one and what is in front of me, not how CF detects legitimate bots.
There’s still a disconnect between the traffic evidence and the error. Cloudflare classifies Google’s bots by using the methods Google has established. If Google is not following their own rules, you’ll have to make an exception for their rogue bot.
https://dnschecker.org/#PTR/34.83.190.242
I suggest you watch your logs to see if you can narrow down the bot traffic that gets through once you make that exception.
In that case you are suggesting that is an IP address of Google. If it is, it is not properly advertised - as I posted in the first response - and blocked for that reason. You’d need to talk to Google for that.
1 Like
I’m not going to talk to Google for that! Whatever is behind this, it’s something between CF and Google. Soon other CF customers will start to complain about this and they have to do something!
Well, so far you are the only one claiming that, so we can’t really tell if that is Google. The response code you mentioned would not suggest it.
In any case, if that really is Google, then that’s something for Google to fix as that IP address is clearly not properly configured and hence can’t be recognised as such.
1 Like
In any case I’m CF’s customer and I pay them to deliver! And if they fail to do so I have my own ways 
The person who is getting paid to chase and fix these kind of issues is definitely not me 