Enhancing TTFB: Google penalized my website after starting to use CF

Website, Application, Performance
1

Hi

I’m aware that this an old question for a lot of CF users, but I’d like to share my worries with you.

I started to use CF some months ago, just because I suffered a DDoS attack in my website, and I decided to activate CF. I struggled to set it up (problems with the SSL certificates), but I managed to launch it. However, at the same day I started to use CF, the value of “Time Spent Downloading a page” of Google Search Console increased automatically from 150 ms to 700 ms.

It’s a huge rise, and after a few weeks, my SEO ranking in Google started to decrease. My website hosts more than 1,500,000 pages, and I suspect that such increase of ‘Spent Time’ is penalized by Google.

As mentioned, I’d like to share my worries with you:

  • I’m not sure if Cloudflare is for me. I was just looking for a DDos protection for my website, but I’ve found that I’ve lost 40% of my web traffic due to the Google ranking penalty.
  • I’ve read the famous post about “Stop worrying about Time To First Byte (TTFB)” (Stop worrying about Time To First Byte (TTFB)). But the fact is that the ‘Spent Time’ by Googlebot crawling my site is much higher now. I’ve just done some performance tests with the same HTML code in two different domains, the first behind Cloudflare and the second without Cloudflare, but in the same server. I’ve done the tests with ByteCheck+Pingdom+GTMetrix. Behind CF: the waiting time is average 350 ms. Without CF: the waiting time is just 70 ms.
  • Honestly, I do not know very well how to configure Cloudflare to optimize the performance. I checked Auto Minify for JS+CSS+HTML; I activated Brotli, Enhanced HTTP/2 Prioritization, and Mirage; I did not activate Rocket Loader; the Caching Level is Standard; the Browser Cache TTL is 1 year; ‘Always Online’ option is activated.
  • I have SSL with Cloudflare (the SSL/TLS encryption mode is Full). Initially, I was also using https before using Cloudflare, and the SSL certificates are also installed in the origin server. I’m not sure if I must remove them from the origin server.
  • I’ve read about Argo and sounds good. But it is a usage-based product with a cost for me, and I’m not sure if I can enhance the speed by using other options.

Any tip would be very welcome. Thank you very much.

2

Hi @motop,

We don’t know your domain, and I understand that having gone through a DDoS you may prefer to not share it, but some suggestions would have to be based on the type of content your website has. Also, it would be good to know which plan you are in (or can afford to obtain), as some features vary depending on plan level.

If your site has lots of static content, you could benefit greatly from caching HTML pages at the edge, by setting a page rule with Cache Level: Cache Everything and Edge Cache TTL. This could be the most impacting of any changes available to you on Cloudflare.

For non-static content, you may want to try Railgun (requires business plan), which will speed up the delivery of non-dynamic content that is part of dynamic pages. It’s available at the Speed tab > Optimization.

AFAIK, any cloud-based service that offers DDoS protection will have to create a layer between your website and your visitors and that does mean an extra hop. Therefore there will always be some time added to TTFB. So, unless you can find a non-cloud solution to the DDoS threat, you should consider the extra time as part of the price to pay for the protection you want.

1 Like
3

This would need to be investigated at your origin end as by default Cloudflare doesn’t cache dynamic HTML generated pages i.e. php generated HTML pages and passes the request back to origin server. CF only caches static assets out of box https://support.cloudflare.com/hc/en-us/articles/200172516-Which-file-extensions-does-Cloudflare-cache-for-static-content-. So time spent downloading a page is usually from your origin unless you have setup specific CF page rules to ‘cache everything’.

From what I have read so far, GSC crawl time for downloading a page isn’t a SEO ranking factor at all. Google uses page speed as reported by new GSC Speed report Core Web Vitals report - Search Console Help and PageSpeed Insights field/origin data https://developers.google.com/speed/pagespeed/insights which is all based on Google Chrome User Experience report (CruX) which is what PageSpeed Insights reports for field and origin data. So I’d verify the same slower page speed with these other 2 tools as well to see what your page speed state is.

Found one article at Google Search Console Crawl Stats For Time Spent Downloading A Page Is The HTTP Request

John Mueller said it has nothing to do with how long the page takes to render, but rather a simple HTTP request time. Basically, how long it takes GoogleBot to complete the HTTP request.

John wrote, “that’s just time to complete a HTTP request, nothing related to rendering the page itself.”

So this does not measure page speed at all.

I’d check out webpagetest.org as they have alot of geographic test locations and allow you to test real mobile devices and different browser clients and connection speeds. I wrote a guide for my users which maybe useful to you as well https://community.centminmod.com/threads/how-to-use-webpagetest-org-for-page-load-speed-testing.13859/

Optimal Cloudflare configuration involves optimising front end visitor experience = Visitors hitting CF datacenter and all that entails + optimal origin connection configuration = the connection between Cloudflare and your origin server.

With Cloudflare Full SSL enabled, optimal CF edge server to origin server connections is when your origin supports TLSv1.3 protocol and uses ECC 256bit ECDSA SSL certificates instead of the traditional RSA 2048bit SSL certificates.

ECDSA SSL ciphers/certificates are more performant than RSA 2048bit cipher/ssl certificates and due to their size, means less data transferred over the network. Also ECDSA performance is also dependent on what crypto library and version your origin web server is built with. For example if Apache or Nginx is built with OpenSSL 1.1.1, ECDSA performance can be up to 40% faster than OpenSSL 1.1.0/1.0.2. OpenSSL 1.1.0/1.0.2 in turn can be up to 2x times faster than LibreSSL crypto library. And the more exotic Google BoringSSL crypto library can have between 10-25% better ECDSA performance than OpenSSL 1.1.1.

TLSv1.3 on origin means CF edge servers can communicate over TLSv1.3 instead of TLSv1.2 to your origin saving your 1-RTT = round trip time which on CF to origin connections can be nice boost given CF to origin communication is still over HTTP/1.1 and not HTTP/2 so connections are LESS multiplexed. See my write up for my members about this at https://community.centminmod.com/threads/improving-cloudflare-connections-to-origin-server-use-ecdsa-ssl-certs.14817/

If you’re using Nginx web server at origin, you can configure it to log what SSL cipher and SSL protocol your origin is serving/connecting with Cloudflare edge servers - example at https://community.centminmod.com/threads/cloudflare-custom-nginx-logging.14790/

3 Likes
4

@motop if you want to PM me your domain I can take a look at your performance.

5

Hi @cbrandt,

Thank you very much for your kind answer. Please let me explain some facts about my website. First of all, I’d like to clarify that I don’t have any experience with Cloudflare. I am an Apache+PHP+mySQL programmer, and have experience with HTML and JavaScript.

  • The content of my website is totally public, and dealing with information on different kind of sport teams around the world. There is no private data, and every page shows the same content for each visitor.
  • The data are stored in a mySQL DB, and the HTML pages are created with PHP through an Apache server.
  • I subsbribed the Pro Plan, plus 1 custom dedicated certificate.
  • As you hint, all of my content is static, and I update it twice a year. As mentioned, I populate the HTML code from the mySQL data, but each URL shows the same contents for six months.

My big question is: Which is the option for “caching HTML pages at the edge”? I thought that Cloudflare was already caching my contents through the ‘Caching’ section of the Dashboard. As mentioned, I did set the ‘Caching Level’ to ‘Standard’. I understood that Cloudflare was displaying my static content, instead of being served form the origin server, but I’m beginning to suspect that I haven’t selected the right option.

How to tell Cloudflare: hey, the content of this URL will not be modified during the next six months. Ask me the contents of this URL once, and then you serve them during the next six months or until I tell you that they have changed.

It makes a lot of sense, thank you again! :slight_smile:

6

Hi @motop, you’re welcome.

For the purpose of what we are talking here, all that matters is the final HTML generated by your origin, whether or not it will be cached by Cloudflare.

Since Cloudflare doesn’t know whether your site is (or may become) dynamic, it doesn’t cache HTML by default. Here’s a guide about what Cloudflare caches by default (hint: static files) and how to add HTML to the equation:

https://support.cloudflare.com/hc/en-us/articles/202775670-How-Do-I-Tell-Cloudflare-What-to-Cache-

Basically, you can keep the Cache Level at Standard in the Dashboard > Cache tab, and create a few page rules to “cache everything”.

Dashboard-level settings apply to the whole zone (all subdomains, all URLs), while page rules set exceptions to the settings you may have set at the Dashboard level.

Go to Dashboard > Page Rules and create a rule with a setting Cache Level: Cache Everything. Add the setting Edge Cache TTL in the same page rule. Make sure you create other page rules to exclude from CF cache the HTML of your back end. At the end, you could have something like this:

PR1

URL: example.com/back-end/*
Settings: any settings will do in terms of avoiding caching HTML, as the position of this rule will prevent the following rule to trigger for the /back-end/ path
Ex.: 
Security Level: High 
Disable Apps
etc

PR2

URL: example.com/*  (include the * before domain if you want the cache to apply to subdomains)
Settings: 
Cache Level: Cache Everything
Edge Cache TTL: 1 month (this is the amount of time cache will remain at the "edge" = each of Cloudflare's data center that have received requests for your domain)
Browser Cache TTL: 1 day (optional, CF will use whichever is larger, your origin or this setting. Even if the content isn't supposed to change for 6 months, I wouldn't set this for more than a day, as you never know when you may find an error in a page that you'd want to fix. Edge cache you can purge anytime, but not browser cache)

Not only will the caching of HTML help with speeding up your pages, but it will also help in terms of DDoS protection, as most requests will be directly served by CF and more origin resources will be freed to handle whatever attack your site may face.

1 Like
7

This topic was automatically closed after 30 days. New replies are no longer allowed.