Is cloudflare Anti SEO or more precise Anti Google?

Cloudflare is basically a Pull CDN. Once the user from a particular location get the content in EDGE and next user will get that content from that Edge, until LRU Algo Hit which will delete that content.

Now my user is not primarily from Mountain view (Google crawling location). So when google crawler come to check my content (crawl). My content is not cached in that particular Edge location. So when google crawl it find my site double slow. First it request content from cloudflare → cloudflare than get that content from server (http 1.1) - > server process the content zip it sent it to cloudflare and cloudflare process that content and sent it to Google. (Basically not cached content and server response will be double slow, my server is in USA.)

During this time my page First Contentful Paint, Largest Contentful Paint, Cumulative Layout Shift all become bad Because of first time access. Even Image will not be optimized as webp or polish optimized for the first time user on that edge location. Even I get Reduce Server Response Time.

Google will not visit that page again in near future. LRU Hit and page get deleted from edge cache in few hours.

If it again crawl the page in next few minutes, I might have best of score possible. But why would it crawl page again so soon?

Now what should I do about that?

Google does not only crawl out of Mountain View. Chances are Google will fetch your pages from their location that’s nearest your origin.

2 Likes

My question still stand. Almost no data center in ASIA and Africa, Australia. Suppose my users are from India, Indonesia, Malaysia, Philippines , UAE, Saudi Arab, South Africa, New Zealand and very few from USA and Canada, cloudflare will cached content most probably in their native country. Google will fetch content from USA or Europe (primarily USA as shown in cloudflare Analytics DNS request)

I think the question in the subject should be rephrased from “Is cloudflare Anti SEO or more precise Anti Google?” to something like “Will Cloudflare negatively impact my search engine rankings?”

3 Likes

As far as I understand, you are concerned about your SEO score for webpages/URLs regarding the end-result of the Page Speed output (or Google Search Console saying your webpages/URLs are slow)?

Regarding the (re)crawl and (re)indexing, I believe that is mostly in the Google’s hands about how often it does, despite the fact we can “control it” via some attributes like priority, lastmod and changefreq for each item (URL) in our physical sitemap.xml file using robots.txt file to either allow Google to use it.

sitemap file:

<?xml ...>
<urlset ...>
...

<url>
<loc>YOUR_URL_WEBPAGE</loc>
<lastmod>ISO 8601 date format</lastmod>
<changefreq>always</changefreq>
<priority>1.0</priority>
</url>

...
</urlset>

Robots file:

User-agent: *
Sitemap: https://yourdomain.com/sitemap.xml

If you want your content to be cached for a longer time, you can setup the Edge Cache TTL and Browser Cache TTL to some higher values at Cloudflare dashboard.

Furthermore, as Google has got servers around the World, maybe the “main” ones are in USA, so all the content get’s crawled, indexed or scraped from there mainly.

  • I know I saw some Google bots and requests from Google in Hong Kong at Cloudflare Firewall events, or either BingBot from different locations … but not sure it they were the legal ones.

As I am not a Cloudflare employee, I really cannot advocate Cloudflare and admit the truth of the title of this topic, and as far as I am using Cloudflare as a customer I can say Cloudflare is not an Anti SEO and not an Anti Google.

Freitasm,

I am already doing this sitemap thing and also increasing crawling speed. (I always increase it in every six month.)

Regarding Edge TTL, LRU Algo always take precedence. Even you set Edge TTL higher.

It is also a nice title but I think mine suits more.

To answer your question in the thread-title:
No. Cloudflare is not Anti Google, as this behavior affects ALL and not only pages on Google. Cloudflare also is not Anti SEO, as it does not slow down your site. Your origin is slow and thats your main problem. So they actually improve SEO for your site, just not most of the time and your are left with how fast your origin is.

Correct!

In my tests even in some minutes but if you request a URL from the same POP often enough it will be cached longer.

Well, that’s first a problem on the origin. If your origin is slow, your page basically is slow. Cloudflare does not and never had a solution for slow sites. What Cloudflare as a reverse proxy CDN offers is a way to conceal the real problem instead of solving it.

Why you rent a server in the states if you already know your users do mainly come from:

?
This for me does not make sense. Longer distance from the origin = higher TTFB.

To be straight forward and honest:
Don’t use Cloudflare if you rely on anything that needs guaranteed performance. Cloudflare does not offer something like that as:

  • the first call will always be slow
  • they purge your cache whenever they want
  • you can not propagate NOR control your cache

But it is for free, so actually you should not expect too much if you are on a free plan. Performance anyway will not improve with paid plans, but on paid plans your Cache is not as often purged like on free plans. I actually lately often spoke about exactly that problem (for example with @amayorga) and developed a solution for its core problem, but it’s hard for me to reach out to people from Cloudflare which actually are in the right position and could listen to me, to judge if the solution I have developed fits their market. That’s a pity, but that’s how it is.

Just to spoiler a little: the main goal of my solution solves the problem with slow

  • websites
  • web shops
  • industrial web applications

entirely and would also fix the problem you have completely by core!

1 Like

You are assuming that my website/server is slow. And also assuming that I am using free cloudflare. Assumption is bad my friend.

Some thing you can not optimize at server level with cloudflare like webp/image optimization. It will create issue with cloudflare and some older browser.

Even if you optimize all your html pages at server, Cloudflare will get that page for first time and process it, google will crawl page 1 time probably in 1 day and LRU will delete it probably that day. My server is in USA thats the primary reason I m using CDN.

Why do you think my site is slow? When I use pagespeed insight and run it first time it has a bad score of 60 on mobile and on desktop 80. But second time check always yield 75+ score on mobile and 95+ score on desktop.

But for google it becomes a bad score because they only access/crawl page once per day. And always get twice slow page. Server Process + cloudflare process. Cloudflare access site over http 1.1 so nothing I can do to improve even if I my site or server support http/2. For first time images are not optimized (can’t optimize at server because of cloudflare restriction). TTFB will be much higher for first time because of two server access.

I am not saying my site is slow for my user. It is super duper fast. I am saying it is slow for google because google gets page which are processed twice by two server on every page.

That’s what you implicated told, because if your site is slow without Cloudflare, or without getting served from Cloudflare’s Cache then yes 100% your origin/site is slow.

Never did so, I just told you about the special case with Cache purge on free plans and how it differentiates from paid plans. And also referred to the fact that some things never change even if you are on paid plan, like “the first request will be slow forever”.

That’s wrong. And the way you are referring to of implementing this (with rewrite rules) anyway is a bad idea.

If done right it will never cause any issue even with:

  • older devices
  • mobile devices
  • older browsers (like IE)

Yeah, but why? If your visitors are not from America?

First time = measures speed of your site delivered by your origin. Again pointing to a slow server

Second time = measures speed of your site delivered/accelerated by Cloudflare’s Edge Cache.

Like I said: if done right on your origin, it will harmonize with Cloudflare perfectly.
HINT: LINK

If your origin would be somewhere where your visitors are, it would really reduce the TTFB.

I understood what you said. Like I said, Cloudflare does not offer a real solution for such a requirement. But you can conceal the problem with Cache-Everything and a high Edge-Cache TTL.

Pointless to argue with you.

You can not optimize webp at server level if you are using cloudflare(and want to cache image at cloudflare). You can not use Brotli at server if you are using cloudflare. You can not use HTTP/2 at server if you are using cloudflare and blah blah.

I am not saying TTFB is slow for my user. I am saying it is slow for Google. I don’t think you even understand what exactly I am saying. For my user website is super duper fast (avg speed score of 1.5 second in last 2 month).

My line of question is it is slow for google and hamper my search ranking.

I think you need to learn about cloudflare LRU before suggesting High Edge TTL. I do not have user in Mountainview and google will read page at max once per day. LRU will delete that page from edge cache no matter what in one day.

Every page google access (please forget about user and there experience for sometime) they get processed twice everytime. No matter how optimize my website is, it will be slow for Google crawl always.

If you can not do something this is showing your limitations, but this never proves that it is not possible. I have done so and converted AND optimized WEBP on serverlevel and it works perfectly.

I agree with you :slight_smile: have a good day!

Every good argument, which is not in favour of cloudflare goes like that. Some of cloudflare hard core fans (I am also one of them as well), creates ruckus and stop the argument which is valid.

AM I WRONG about how cloudflare behaves with google? Is there anything wrong in logic?

Im think the problem here is about the assumptions made, not the actual question.

First, lets break down the problem:

Acting as “pull CDN” is the first request for the main server be will be slower than direct by the server?

You assumption here is “twice work” and “twice slow”… But thats does not make sense. Cloudflare will stream the request, not download->cache->serve separately, thats is done in parallel. If any overhead exists will be in the range of 1-10% max.

You can just test that, disable the Cloudflare proxy, check your score, latency, etc…

In reality, under Cloudflare proxy the first request can be faster. Because Cloudflare will talk with ou origin server, not necessarily via public internet, but under Cloudflare internal/private network. Ever if you host does not have any interlink with Cloudflare, (and even without Argo smart router) you can expected to Cloudflare call from the nearest COLO to you server.

If you live on South Korea for example and you server in on US, the traffic maybe inside of Cloudflare until very close to you server, and exit in the same datacenter of very close.

Cloudflare can, and may/may not chose to cache the request in the US colo before stream to you SK colo, and will make any user (including google) to get a cached content in US too. But even if they not do that, if you account the “better” route that you got, the final “slow” should be at max 1%

The second point is:

Will google mensure my site score based on this first request ever?

Well… Google is not dumb, they know the request is behind Cloudflare, they know how Cloudflare works, and they can see on the response headers if they got a cache miss ou anything like that.

They probably will adjust the score accordingly, and even without knowledge of the Google “black box”, you can expect to get a better score from Google using Cloudflare because they know how any pull-cdn works, and after see the “cache miss” they can know how the next request will better because they have good stats on the performance from all CNDs.

So… In resume: Cloudflare is “Pro SEO” and “Pro Google”, because Google are not dumb and know how to score a site behind of a reverse proxy.

1 Like

My first assumption about double request is based on a result. When we do pagespeed insights first request always show. Assuming this happens to google crawler as well as I do not have real user in those colo.

Second request remove those errors. Just 100ms improvement to mobile page loads boosts an eCommerce website’s conversion rate by 8.4%.

Google do not choose difference between company. They have set a criteria. Result is shown in search console. According to those result they are definitely not giving any preference to cloudflare and the first time result are only considered.

Well, try to keep the balance and just optimize for people, not for search engines :wink:

You are over looking Google Page Insights. This free tool is just a small percentage of how Google scores any site.

Page Insights just does a small subset of testing from Google Lighthouse for example. Lighthouse does indeed several requests for you site. But ever that, is just a small subset of all the analyses the “real” Google does for score your site.

Do you generally think the “google boot” test all sites in the world from mountain view and scores then based on the latency that they have to they HQ?

Google Page Insights is just that… a Insight of your site. Thats now how Google actually scores you site.

If you really look of the the location of all boots (several bots) that google boot send to you (any) site, you will see they are from everywhere. And each one does different kinds os request, etc…

Like a I said before: Google Knows know pull CDN works. They have they own pull CDN too… They not only can, but actually look of all response headers, and they know when they got a cache miss.

@least cloudflare analytics suggest that google does access my site from USA only.

Search engine brings users which brings money.