Achieving 7 Day TTL along with Revalidation

Hi, We would like to have the Cloudflare Data Center hold the cache for at least 7 days and perform revalidation with server for every request. For this, we followed this article (Cache with no-cache headers) and tried ‘public, s-maxage=0’ & ‘public, no-cache, proxy-revalidate’ cache-control headers with Origin Cache Control ON. We are seeing the cache invalidation/revalidation to work but the content is expiring in the Cloudflare Data Center between 1-12hrs. We are seeing the cache lifetime to be more when using ‘public, s-maxage=60, proxy-revalidate’ cache control header but still less than 12 hrs. Can you suggest how we can avoid content expiry in Cloudflare Data Center (at least for 7 days) and also have the revalidation with origin server happening for all of the requests during this 7 days timeframe. Thanks.

Users can’t force Cloudflare to retain cached content for any period of time. You can politely ask, but if Cloudflare deems that content to be low priority, it will be evicted from the cache. Higher tier plans do get cache priority, but, again, there are no guarantees. Especially for a full week.

If you want a CDN to respect your TTL, you might find a traditional CDN that does this.

1 Like

Thanks @sdayman,
Can you suggest if there are any common workaround strategies to achieve this(or close to this) as this seems to be a common requirement to us. We would like to achieve 7 day TTL and have revalidation happening for every request. We have procured Cloudflare Enterprise and have started using few features heavily, it might be hard for us to switch to another CDN.
Thanks

You’ve already got this part down, but there’s just no way to guarantee the 7 day cache. If a static file is called once, even then it might go to cache and take a second time. And this would have to be through the same edge server, as all edge servers are independent. I didn’t mention that bit.

You’d really be better off with a dedicated CDN with more flexible options.

If you are on CF Enterprise plan then you have a solution in terms of pre-warming your CF CDN cache on all 200+ CF datacenters using CF Cache Prefetching https://support.cloudflare.com/hc/en-us/articles/206776707

Does Cloudflare Do Prefetching?

Yes. Enterprise customers can enable prefetching by turning on ‘Prefetch URLs’ in the Speed app of the Cloudflare dashboard, and including a list of URLs to prefetch in HTML headers.

What is Prefetching?

Prefetching means that Cloudflare pre-populates the cache with content a visitor is likely to request next. This guarantees a higher cache hit rate and thus a faster experience for the user.

I’m using CF Cache Prefetch and it works nicely. However, just be aware this will drive up your CF request count and increase CF Argo bandwidth costs (if Argo is enabled) and increase CF Worker requests (if CF Workers are enabled on your zone). This is because all 200+ datacenters will regularly hit your origin assets/pages defined in your CF cache prefetch manifest file to pre-warm your cache. The increase depending on what/how many assets you put into your manifest file can be as much as 5x to 30x fold increase. So if you were previously doing 1 million CF requests/day - with CF cache prefetch enabled you could be doing anywhere between 5 - 30 million CF requests/day now.

You can verify CF cache prefetch is working by inspecting your origin server’s logs for user agent = CloudFlare-Prefetch or with CF Enterprise inspect your logpush logs.

Example I wrote a parser to parse my CF Enterprise logpush logs and here’s a filter of today’s user agent = cloudflare-prefetch for specific static js file I have in my CF prefetch manifest file which tells CF edge servers to prefetch and warm the cache for this asset = domain.com/filename-5.1.0.js and return summary results including my CF Bot Management score and how the score was derived.

You can see each line as summary for each CF datacenter IATA code and also the coloid as well.

this is just the top 25 CF datacenters summary filtered

./cflog-parser.sh parse domain.com/filename-5.1.0.js allips 20200917 100 none cloudflare-prefetch | awk '{print $5,$6,$11,$24,$18,$20,$21}' | sort -n | uniq -c | sort -rn | head -n25
  19822 /filename-5.1.0.js HEAD IAD-coloid-16 CloudFlare-Prefetch/0.1; 6 Machine Learning
  10535 /filename-5.1.0.js HEAD FRA-coloid-71 CloudFlare-Prefetch/0.1; 6 Machine Learning
   7072 /filename-5.1.0.js HEAD AMS-coloid-20 CloudFlare-Prefetch/0.1; 6 Machine Learning
   4056 /filename-5.1.0.js HEAD CDG-coloid-19 CloudFlare-Prefetch/0.1; 6 Machine Learning
   3234 /filename-5.1.0.js HEAD SJC-coloid-4 CloudFlare-Prefetch/0.1; 6 Machine Learning
   2910 /filename-5.1.0.js HEAD SIN-coloid-35 CloudFlare-Prefetch/0.1; 6 Machine Learning
   2406 /filename-5.1.0.js HEAD LHR-coloid-21 CloudFlare-Prefetch/0.1; 6 Machine Learning
   1840 /filename-5.1.0.js HEAD SEA-coloid-28 CloudFlare-Prefetch/0.1; 6 Machine Learning
   1793 /filename-5.1.0.js HEAD NRT-coloid-22 CloudFlare-Prefetch/0.1; 6 Machine Learning
   1792 /filename-5.1.0.js HEAD SYD-coloid-26 CloudFlare-Prefetch/0.1; 6 Machine Learning
   1548 /filename-5.1.0.js HEAD ARN-coloid-128 CloudFlare-Prefetch/0.1; 6 Machine Learning
   1522 /filename-5.1.0.js HEAD HEL-coloid-79 CloudFlare-Prefetch/0.1; 6 Machine Learning
   1458 /filename-5.1.0.js HEAD LAX-coloid-12 CloudFlare-Prefetch/0.1; 6 Machine Learning
   1404 /filename-5.1.0.js HEAD YVR-coloid-69 CloudFlare-Prefetch/0.1; 6 Machine Learning
    974 /filename-5.1.0.js HEAD BOM-coloid-202 CloudFlare-Prefetch/0.1; 6 Machine Learning
    954 /filename-5.1.0.js HEAD EWR-coloid-11 CloudFlare-Prefetch/0.1; 6 Machine Learning
    910 /filename-5.1.0.js HEAD DME-coloid-87 CloudFlare-Prefetch/0.1; 6 Machine Learning
    764 /filename-5.1.0.js HEAD DFW-coloid-15 CloudFlare-Prefetch/0.1; 6 Machine Learning
    344 /filename-5.1.0.js HEAD YUL-coloid-70 CloudFlare-Prefetch/0.1; 6 Machine Learning
    266 /filename-5.1.0.js HEAD SLC-coloid-143 CloudFlare-Prefetch/0.1; 6 Machine Learning
    234 /filename-5.1.0.js HEAD HKG-coloid-23 CloudFlare-Prefetch/0.1; 6 Machine Learning
    160 /filename-5.1.0.js HEAD KBP-coloid-132 CloudFlare-Prefetch/0.1; 6 Machine Learning
    126 /filename-5.1.0.js HEAD ICN-coloid-34 CloudFlare-Prefetch/0.1; 6 Machine Learning
    112 /filename-5.1.0.js HEAD ATL-coloid-27 CloudFlare-Prefetch/0.1; 6 Machine Learning
    102 /filename-5.1.0.js HEAD YYZ-coloid-29 CloudFlare-Prefetch/0.1; 6 Machine Learning
2 Likes

Thanks @eva2000,
We have 200 urls that we are looking to be warmed up always. If we mention them in the manifest file, will all of those URLs be requested to our origin server (by a Cloudflare data-center) for every page request from user?

Sort of yes. From what I gather based on the log parsed data you see there’s an uneven number of requests from some CF datacenters. So logic would dictate it’s based somewhat on visitor requests and which CF datacenter they hit. Not 100% sure though.

I generally just prefetch the static assets which generate and render the page’s above fold critical render path. This way you get best page speed in terms of Google’s Core Web Vital metrics. Without needlessly driving up origin request traffic/bandwidth for entire HTML pages.