Google is not crawling my website if i enable the cloudflare on my website

Google is not crawling my website if I enable Cloudflare on my website.

I am seeing a huge traffic drop in the last few weeks and today I realized that Google is not fetching my



URLs or not indexing my edited posts. and showing me this error if submit any URL in Search console or in Mobile-friendly test:

Something went wrong

If the issue persists, try again in a few hours

Disabling the Cloudflare (enabled Development mode) fixed the issue and everything came to normal, so I must say this is purely related to Cloudflare.

Screenshot attached.

Tool Where I am testing:
https://search.google.com/test/mobile-friendly

URL to test:
https://bestchairsreviews.com/best-gaming-chairs-under-150/

One more thing: Created a ticket on Cloudflare support and till today they didn’t fix the issue and they’re taking days to reply.

If Google crawler is being blocked by a Cloudflare security feature, there should be a log of it at Firewall > Overview.

1 Like


Past 24 hours logs are clear but these are the logs from the 22 Feb.

On that page, click on “Add filter”, pick field “User-Agent”, then “contains”, then “Google”.

It should return with issues affecting the Google crawler (as well as perhaps some bots pretending to be Google)

1 Like

Thanks i just did that, but nothing showing…

That means no security feature of Cloudflare is blocking Google.

One possibility is that your hosting provider may be rate-limiting Cloudflare. Please read 522 and talk to your hosting provider’s support to make sure they are not firewalling or rate-limiting Cloudflare IPs.

1 Like

Ok, I contacted them and attached the reply from them.

also, I tried disabling all of my plugins including caching and security plugins. and cleared cache from Cloudflare but the issue still persists.

Cloudflare is not blocking your page. If it were, GSC would display it as a regular error on its Coverage panel. It is only displaying the error when you test with the test tool. That makes me think that perhaps critical assets (CSS, JS) requested to render the page are somehow either blocked or scrambled.

Are you using any CDN other than Cloudflare to host these files? If so, try to disable the CDN and serve all the files locally, see if that makes a difference. See this thread on Twitter: https://twitter.com/sambuno/status/1462629307637977095

Another possibility is some odd combination of Cloudflare features. Do you have Rocket Loader on? Try disabling it. Check any feature you might have recently enabled, by going to your Cloudflare Dashboard > Audit Logs.

Another @MVP may have a better clue. I remember @M4rt1n has recently escalated a similar issue, not specifically related to the Mobile Friendly test, but to Google Search in general, IIRC. I’m not the issue was solved.

4 Likes

In the meantime, the IP address from the screenshot is from Cloudflare network.

I’d suggest to restore original visitor IP as follows on the below article:

Despite they sa “RunCloud” is not enabled for your website, kindly, I’d also suggest to ask RunCloud and your hosting provider to re-check if Cloudflare IPs are allowed to connect/pass to the origin host/server:

Nevertheless, Cloudflare IP addresses list can be found here:

And I am not sure why Googlebot uses HTTP 1/0 in this “mobile-friendly test”.
Can you ask them to share what is the user-agent of it? :thinking:

In terms of a Firewall Events, I’d suggest filtering all the results by the AS number 15169 (AS15169).

2 Likes

No i am not any other CDN, just using Cloudflare and a caching plugin Wp rocket.

also I tried disabling the wp rocket but that also didn’t fixed the issue. it’s still disabled so you can even check.

Kindly, may I ask you to follow the instructions how to propperly setup WP Rocket while using Cloudflare as follows on below articles:

That’s not an issue, i already tried disabling all of my plugins.

not even a single plugin was active when I run a test.

Not quite.

What is the script type="rocketlazyloadscript" for a type MIME-type?
That’s maybe why you have some issues on Google as Googlebot doesn’t recognize it.

And I still se resources being served from /wp-content/cache/.

You also have another cache plugin, aside WP Rocket - x-nginx-cache: WordPress.

And another one x-endurance-cache-level:0.

And maybe using Page Rule like Cache Level: Cache Everyting as far as the cf-cache-status: REVALIDATED (should be DYNAMIC)?

I’d suggest disable each.
Clear cache at your origin WordPress + cPanel (endourance?).
If using Page Rule as above mentioned “Cache Everything” disable it.
Click on the “Purge Everything” at Cloudflare dashboard → Caching → Configuration.
Leave if for few hours so we could re-check and see if anything different.

3 Likes

All the plugins are now disabled, please check.

Endurance Cache from Wordpress is at LEVEL 0

Nice, thank you for feedback :+1:

And now the tested URL passed the “mobile-friendly” test :slight_smile: , see below:

1 Like

And images now failed to load as far as the resource path is being “stated” in data-lazy-load attribute due to the Plugin instead of the img src="", which has got some “temporary image”.

Rather setup loading="lazy" attribute to each img tag than using that plugin.

Please be aware and check above articles related to the WP Rocket if you are obviously using it and configure it as it should be while using Cloudflare, and as far as I saw the style type="" was also some “rocketlazyload” which is incorrect type for CSS and JS code.

And caching WordPress articles/pages as a .html is okay if done at the origin host, but not good if you are using a Page Rule Cache Everything within it as far as visitors may encouter issues as seeing a admin bar at the top from some “cached logged-in user”.

I am afraid that’s the issue in your case here.

2 Likes

Thankyou so much, I activated all the plugins back and issue is still resolved.

I think some feature from Cloudflare is causing the issue because I disabled all featured, security, Speed, caching etc. but now I’ll activate all the options 1 by 1 and see which feature was causing the problem with the website and update you here.

Again, thankyou for all your help.

1 Like

If so, my best gues would be and I’d suggest yo tu disable Rocket Loader and Auto Minify options.

But please check above articles due to the WP Rocket.

Set Cache Level to “Standard” and Browser Cache TTL to “Respect Existing Headers”.

2 Likes

Ok It’s done, and disabled the wp rocket loader and cache everything rule. will check and do your other suggestions.

Please let me know if you’ve any other suggestions.

also thankyou @cbrandt for your help too and for connecting me to the right person.

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.