Cloudflare CDN increase download time of pages by Googlebot from 0.6s to 1.3s

Website, Application, Performance
, ,
1

Good day. Recently I started using Cloudflare CDN for my site. And I see on “Crawl stats” form that “Time spent downloading a page” is increased. I made test using “Fetch as Google” tool, the result you can see on screenshot. Download time with Cloudflare is 1.2s, without 0.6s.

download..PNG2226×1004 154 KB

I am expecting that using CDN should decrease download time or at least leaves the same. But here it is increased twice.

Do anyone else use Cloudflare and have solved this problem?

P.S. I have already wrote ticket to Cloudflare support but do not getting answer.

1 Like
2

How recently did you set up CF? It will take some time before you notice speed increases.

Also, you may like to take a look at
https://support.cloudflare.com/hc/en-us/articles/200172856-My-website-is-slow-or-having-performance-issues

1 Like
3

I set up CF about 2 weeks ago. Please hear me - I have performance issue wit Googlebit. Only with Googlebot.

1 Like
4

can I ask where is your origin server located?

1 Like
5

Server located in datacenter of provider “Selectel” in Moscow, Russia.

1 Like
6

well I am not expert but probably something like this happen:

with cf:
google bot usa->cf usa pop->russia origin

without cf:
google bot russia->russia origin

you can see that without cf the requests actually going through faster routes to your origin and therefore faster google bot response times

1 Like
7

Yes, I understand it, but it’s not ok. Googlebot is the most important visitor for all sites and if Cloudflare increases response time in two times for this visitor then noone wants to use CF. But a lot of people want and actually use CF, so I guess the problem is in particular configuration.

1 Like
8

Just for more information, do you have the IP address of the Googlebot with the faster load times? It should up in your server’s access logs.

1 Like
9

66.249.66.202

1 Like
10

Well…that’s not as helpful as I’d hoped. With Anycast, it could be anywhere. Now I’m wondering how Google decides where the best place to crawl from is.

What do the page load times look like from Sucuri?
https://performance.sucuri.net

I know you said the issue is with Googlebot, so it’s a mystery to me how Google decides the best way to crawl a site and why your load times changed.

Does Googlebot provide a waterfall view of the crawl? Something that would show TTFB and actual load time?

1 Like
11

they decide by the origin location. if they see your server is in europe they will crawl it with google bot europe… of course with Cloudflare they cant know so they fallback to US default

1 Like
12

https://performance.sucuri.net does not accept IDN domain names even in punnycode.
Googlebot does not provide any waterfall view.

1 Like
13

But I am not the first one Cloudflare’s client from Europe :slight_smile: Others should face the same problem and the problem was somehow solved.

1 Like
14

Maybe some of our Euro leaders have an idea of how Googlebot crawls their sites. @Matteo or @sandro?

1 Like
15

Not really unfortunately. The origin I have most stats with is on Firebase Hosting, which is Google itself (ok Fastly, actually, but still billed through Google), with a CDN backed in, then passing through CF which improbes things a bit but won’t care much from where Google crawls since there are origins everywhere.

2 Likes
16

:blush: I am afraid I keep Google at a reasonable distance (read: blocked them from crawling :smile:), so I cant comment on that in particular.

However it seems the TTFB is generally around one secondish http://sitemeer.com/#https://фиас.онлайн. In this particular case it seems to dislike the request and closes the connection but the first response arrives after about 1.5 seconds.

2 Likes
17

who said you are the only one? I am also facing this “problem”, I actually thinking of making a mirror origin in usa just for usa requests
I remember seeing more people reporting it

edit: I found for you example when it asked Google Product Forums
and:
Cloudflare might affect your SEO — Powered Cache

1 Like
18

You’re focusing on the wrong test and metric for relation to page speed SEO with Google and CDNs. Test with webpagetest.org and google page speed insights which now uses the Chrome User Experience Report (CRuX) which is what Google users for paegspeed measurements now especially the Google Lighthouse Report for mobile 3G https://developers.google.com/speed/pagespeed/insights/. Google bot has no bearing on Google determined pagespeed or SEO/rankings these days.

2 Likes