Cloudflare CDN increase download time of pages by Googlebot from 0.6s to 1.3s

Good day. Recently I started using Cloudflare CDN for my site. And I see on “Crawl stats” form that “Time spent downloading a page” is increased. I made test using “Fetch as Google” tool, the result you can see on screenshot. Download time with Cloudflare is 1.2s, without 0.6s.

I am expecting that using CDN should decrease download time or at least leaves the same. But here it is increased twice.

Do anyone else use Cloudflare and have solved this problem?

P.S. I have already wrote ticket to Cloudflare support but do not getting answer.

1 Like

How recently did you set up CF? It will take some time before you notice speed increases.

Also, you may like to take a look at

1 Like

I set up CF about 2 weeks ago. Please hear me - I have performance issue wit Googlebit. Only with Googlebot.

1 Like

can I ask where is your origin server located?

1 Like

Server located in datacenter of provider “Selectel” in Moscow, Russia.

1 Like

well I am not expert but probably something like this happen:

with cf:
google bot usa->cf usa pop->russia origin

without cf:
google bot russia->russia origin

you can see that without cf the requests actually going through faster routes to your origin and therefore faster google bot response times

1 Like

Yes, I understand it, but it’s not ok. Googlebot is the most important visitor for all sites and if Cloudflare increases response time in two times for this visitor then noone wants to use CF. But a lot of people want and actually use CF, so I guess the problem is in particular configuration.

1 Like

Just for more information, do you have the IP address of the Googlebot with the faster load times? It should up in your server’s access logs.

1 Like

66.249.66.202

1 Like

Well…that’s not as helpful as I’d hoped. With Anycast, it could be anywhere. Now I’m wondering how Google decides where the best place to crawl from is.

What do the page load times look like from Sucuri?
https://performance.sucuri.net

I know you said the issue is with Googlebot, so it’s a mystery to me how Google decides the best way to crawl a site and why your load times changed.

Does Googlebot provide a waterfall view of the crawl? Something that would show TTFB and actual load time?

1 Like

they decide by the origin location. if they see your server is in europe they will crawl it with google bot europe… of course with Cloudflare they cant know so they fallback to US default

1 Like

https://performance.sucuri.net does not accept IDN domain names even in punnycode.
Googlebot does not provide any waterfall view.

1 Like

But I am not the first one Cloudflare’s client from Europe :slight_smile: Others should face the same problem and the problem was somehow solved.

1 Like

Maybe some of our Euro leaders have an idea of how Googlebot crawls their sites. @Matteo or @sandro?

1 Like

Not really unfortunately. The origin I have most stats with is on Firebase Hosting, which is Google itself (ok Fastly, actually, but still billed through Google), with a CDN backed in, then passing through CF which improbes things a bit but won’t care much from where Google crawls since there are origins everywhere.

2 Likes

:blush: I am afraid I keep Google at a reasonable distance (read: blocked them from crawling :smile:), so I cant comment on that in particular.

However it seems the TTFB is generally around one secondish http://sitemeer.com/#https://фиас.онлайн. In this particular case it seems to dislike the request and closes the connection but the first response arrives after about 1.5 seconds.

2 Likes

who said you are the only one? I am also facing this “problem”, I actually thinking of making a mirror origin in usa just for usa requests
I remember seeing more people reporting it

edit: I found for you example when it asked https://productforums.google.com/forum/#!topic/webmasters/secGpj_1O7k;context-place=topicsearchin/webmasters/category$3Acrawling-indexing--ranking|sort:relevance|spell:false
and:
https://poweredcache.com/blog/2017/12/19/Cloudflare-might-affect-seo/

1 Like

You’re focusing on the wrong test and metric for relation to page speed SEO with Google and CDNs. Test with webpagetest.org and google page speed insights which now uses the Chrome User Experience Report (CRuX) which is what Google users for paegspeed measurements now especially the Google Lighthouse Report for mobile 3G https://developers.google.com/speed/pagespeed/insights/. Google bot has no bearing on Google determined pagespeed or SEO/rankings these days.

2 Likes