Googlebot seems being blocked. How can I trace connctions it makes to CF?

Hi
Google bot mobile has difficulties in accessing some assets of my websites, like style sheets. It results in mobile usability errors. I’ve made bunch of tests with third party tools - none of them failed. Everything is working properly. Only google bot mobile is having problems, and only on one of my wesites. I even disabled every security feature in CF (firewall, bot fight mode etc) but it didn’t help.
How can I check if googlebot really tries to connect to CF servers? Like tcpdump… I’ve setup firewall rules to allow googlebot ip ranges - I can see connections being made but usually not every css is being accessed.

have log ?

try it

(cf.client.bot)

allow

Thanks for suggestion
I’ve already allowed google bots ip as published here https://developers.google.com/search/apis/ipranges/googlebot.json
I’ve just added cf.client.bot - let’s see if it helps

Greetings,

Thank you for asking.

If it’s blocked, you’d see the blocked or challenged request from Googlebot in Firewall → Overview tab.

Furthermore, there are fake requests as “Googlebot” user-agent from different addresses which aren’t from the Google AS number, even I saw few times coming from Google AS number but not the correct and not in published IP address range from Google → which you’ve already linked.

with the

Should work.

Thank you for reply.

The problem is that in Google Search Console I’m getting “unknown error” because googlebot could load the resources. If it’s really trying to connect I should see any trace of it in Firewall, but there is no occurrence of Allowed nor Blocked request. It seems that it fails to reach Cloudflare.
I’ve paused CF for my website to route all traffic directly to my server - the errors are still present and I see no connections from google in logs / tcpdump. So… it looks like it’s a Google’s problem or routers between.

I am sorry to hear that.
Unfortunately, definetly not the first case here as far as there are topics with similar issue.

I just don’t understand how this occurs and happens.
On none of my 70+ websites on Cloudflare, I have no issue with “rich snippets” or “mobile-friendly” testing tools, despite “resources not loaded”.

That’s interesting.

I am starting to think what could it be, like it might be due to some JavaScript code which is placed at the top of the <head> HTML element, or some caching plugin if we are using WordPress, etc., otherwise some “beta” features like Crawl Hints / SXG if enabled.

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.