@rommes As a rule of thumb, I’d say if the frequently updated content is a large portion of your site (visually), then SXG is not a good fit. However, if it is small (e.g. sidebar content) then you can use lazyloading JS to add that after the SXG loads. You may want to pre-allocate space for it to avoid hurting your CLS metric. You can also make a page-by-page decision on this using the Cache-Control header.
That said, this is just a rough rule of thumb; a more precise rule may require doing some experimentation. For instance, one thing that affects this decision is how often people view the SXG (at which point the cache may fetch an updated copy). If you create SXGs with a smaller max-age, then they will often be expired by the time somebody visits the page. If you can monitor how often SXG is served, then perhaps you can tune that parameter to make the trade-off between cache hit rate (latency) and freshness. This is just an idea; I don’t know for certain if such an experiment would work (e.g. have high enough signal-to-noise ratio).
@user2890 It may be possible to make such pages compatible with SXG, by creating an initial ajax request for a CSRF token (no ACAO header; same-origin only) as this StackOverflow topic suggests. But this may slow down your non-SXG pages by adding another request to the waterfall. If you can do it on SXG pages only (e.g. using the Accept header) that might be best.
You control SXG cache requirements and freshness & SXG cache time via your cache-control max-age headers set either at Cloudflare dashboard browser cache ttl or your web server origin set cache-control headers you configure yourself.
This helps if you are unable to derive the ASN of the request from your origin server - let Cloudflare do that part via Transform rule and then your origin server can look for the custom tagged request header i.e. X-Googlebot in screenshot.
Thanks for your comment, buddy. Actually, that’s too technical for me. I want to try the SXGs Cache Period of 3 minutes but Cloudflare by default provides either 2 Minutes which causes SXG Error or 5 Minutes which is quite a high value for the type of site I run.
That’s why I am looking forward to the inclusion of SXGs Specific Cache Period from the Cloudflare Team
@firat just noticed Cloudflare’s own blog’s SXG requests are returning cache lifetime too short
Request URL: https://blog-cloudflare-com.webpkgcache.com/doc/-/s/blog.cloudflare.com/automatic-signed-exchanges/
Request Method: GET
Status Code: 200
Remote Address: 22.214.171.124:443
Referrer Policy: origin
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
content-type: text/html; charset=UTF-8
date: Mon, 22 Nov 2021 04:06:07 GMT
warning: 199 - "debug: content has ingestion error: Error fetching resource: Content is not cache-able or cache-able lifetime is too short"
also interesting is if my SXG enabled blog Google search result is a featured snippet it doesn’t get SXG cache prefetched, instead the first regularly search result does? @twifkak so Google Search featured snippets are treated differently than regular results for SXG ?
the first regular search result was Cloudflare’s blog heh which has SXG enabled but isn’t cached due to cache lifetime too short
If my blog is a regular search result, the Google does prefetch and cache SXG version for faster response times
Every time we enable SXG, Avg. Page Download Time grows at least x5 times from the “normal” value.
Furthermore, these spikes happen only for Chrome clients, and if I filter out all browsers except chrome, the difference between the normal value and a spike is almost x20.
I don’t have a clear explanation for why this is happening. Has anyone noticed the same behavior?
Might want to narrow down your traffic to just Android mobile Chrome from Google search/googlebot to investigate.
Remember Google Analytics is real world users so your average download times are dependent on real world user’s ISP speeds and device/browser CPU and capabilities. It only takes an adequate proportion of very slow devices/ISP connections to bring down your average times.
But make sure your SXG cache lifetime is adequate otherwise visits from Google Android Chrome search referrals will most likely be doing a SXG cache miss and redirecting to the non-SXG cached version of your page. You can see 1st post of this thread and my SXG guide at Testing Page Speed With Cloudflare Automatic Signed Exchanges & Google Search Cache - Centmin Mod Blog on how to inspect the headers. Also non-cache non-prefetched SXG is slower than regular CF CDN cached HTML pages - when SXG cache lifetime is too short to be of use on visitor Google Android search from my tests using 1. Webpagetest. The benefit of SXG is really when Google search prefetches and stores your SXG version of your page when visitor uses Google search on Android. If you pages don’t even rank on 1st page of Google search results, then probably less likely to do that SXG cache prefetch as I understand it? @twifkak ?
Google Analytics with custom segments to narrow down traffic I want to insights into.
Avg Page Download Time breakdown - you can clearly see download times for real world are dependent on speed and the type of device used. Desktop is fastest with average device power and speeds much higher. Android mobile without SXG cache slowest as expected and SXG cached Android Google search was 2nd fastest overall and fastest for the segments of mobile traffic that we are looking at.
Pagespeed Insights only measures really above the fold page load versus Avg Page download time would take into account the entire page download so I guess with a lot of images = slower page download times. So the question is do you even care about Avg Page download times when it’s not really being measured by Google user experience page signal like Core Web Vital metrics? SXG is meant to improve CWV metric LCP via improving TTFB.
I might disable CF SXG for this month and see how it compares.
I think there are a few potential benefits to SXG:
Big speed improvement if the SXG is prefetched from Google Search. This currently only occurs for the 1st SXG result of every search results page, but that may evolve over time to best meet user & publisher needs.
Small speed improvement for the other SXGs on the SERP. Because there was a prefetch for webpkgcache.com for 1 result, that TLS connection can be reused for the other SXG results. (It’s as if Google Search had done a <link rel=preconnect> to the result site.)
Connection resilience. Maybe the user has a spotty connection, and was able to prefetch the SXG, and can now still click the link even though their internet is down (with a degraded experience due to missing subresources).
I think, in practice, that first item (prefetch) is the most significant, but I’m eager to see more real-world testing to verify that. It can very much depend on the page – e.g. imagine an app shell where all the content is in non-preloaded subresources. It can also depend on what % of your page views come from Google Search, are on mobile, etc.
I’m just guessing, but it’s possible the Page Download Time increases because it takes Cloudflare time to spin up the worker & generate the signatures. However, this time is only seen by Googlebot, not by users:
If webpkgcache.com doesn’t have a fresh copy of the SXG, it redirects users to the original URL.
When users visit the original URL, Cloudflare serves the unsigned HTML, not the SXG.
I encourage people to look at the impact on client-side metrics like web vitals. Those are the ones SXG aims to improve (especially FCP & LCP), moreso than server-side.
I think the best way to differentiate SXG traffic from non-SXG traffic is by serving slightly different HTML based on the Accept header like I proposed for cache-control. You could use this in Google Analytics to drive a custom dimension, for instance. I’ve proposed a few mechanisms for doing that without any server logic, which I hope to see Cloudflare implement in the future.
Thanks for the details on the benefits. I will see if differentiation can be done at Cloudflare Transform rule level later.
I’ve disabled SXG for this month to see what the differences are. Seems being only 1st ranked search result being prefetched is rather limiting. Hopefully, that is expanded to also include Feature Snippets