How to disallow STALE content being served when origin is down?

First of all I don’t have ‘Always on’ enabled and I am using Argo tunnel to connect to my local machine for testing. Also I thought this would be a simple thing to understand but I can’t seem to find a solution.

Let’s say my origin serves a file /ostrich.png with header cache-control: public, max-age=30.

I’ll get cf-cache-status: HIT response up until when 30 seconds has passed and then it will refresh the resource.

I then take my server completely down.

After 30 seconds I’m seeing cf-cache-status: STALE instead of an expected 5xx origin error message. There’s a 2 second delay as it makes an attempt to connect and fails.

This is the behavior I would want if I had set stale-while-revalidate but I didn’t set that header for this resource. The help says:

STALE:
The resource was served from cache but is expired. Cloudflare couldn’t contact the origin to retrieve the updated resource.

But I don’t want this. I want an error if the server is down after the resource has expired. The behavior I’m seeing is effectively the ‘Always on’ behavior without the banner.

I originally was trying to implement stale-while-revalidate and stale-if-error but without setting either of those options I’m effectively getting an infinite time serving stale resources. What am I missing?

You shouldn’t confuse browser cache control versus cloudflare cdn cache TTL values. Cloudflare CDN cache has a minimum CDN cache TTL based on the CF plan you’re on. So Free = 2hrs, Pro = 1hr, Business = 30min and Enterprise can have 1 second.

From Our Plans | Pricing | Cloudflare

Min Cache TTL Expiry

Cloudflare respects all existing headers, unless you choose to overwrite them. The minimum edge cache expire TTL sets the refetch time for when Cloudflare checks the origin server for a new resource.

So unless you’re on Cloudflare Enterprise plan, setting cache-control max-age will not let you expire CDN cache after 30 seconds. Free plan would only expire CDN cache after 2hrs and browser cache would be either what Cloudflare default browser cache time is or your origin max-age - whichever is longer.

What does the Browser Cache TTL do?

The Browser Cache TTL specifies how long cached files will remain in your visitor’s browser cache. This expiration time is what Cloudflare will set unless longer time periods are specified at your web server.

A longer expiration time ensures faster load times for repeat visitors. However, a longer expiration time also means slower update times if those files are modified.

To learn more about Browser Cache TTL see here.

If you want lower min CDN cache TTL values on Free or Pro plans, you’d need to use Cloudflare Workers and caching via Cache API https://developers.cloudflare.com/workers/runtime-apis/cache & example at https://developers.cloudflare.com/workers/examples/cache-api or via fetch https://developers.cloudflare.com/workers/examples/cache-using-fetch

2 Likes

Thanks for the reply. I think what you’re referring to only applies to the dropdown menu options. I have chosen ‘Respect existing Headers’ which is available on Pro plan.

  • I have a test file that outputs the current time. It has a .png extension to cache by default without needing to enable ‘Cache Everything’.
  • I set the cache control header at my origin to cache-control: public,max-age=20
  • If I load the file on my laptop I’ll see 10:42:00pm from cache
  • I wait 10 seconds and refresh and I see a HIT with the original cached time of 10:42:00pm
  • I also load the page on my phone on a different network altogether. It shows 10:42:00pm
  • I wait another 10 seconds and reload the page from my phone
  • It now shows 10:42:20pm

So that’s a 20 second cache with no minimum of 1 hour being enforced for the Pro plan. It’s just the dropdown that has that limitation.

PS. I am already using workers for more complicated caching, but I was just doing some simple tests on a subdomain. There is no worker, no page rules.

That is browser cache not CDN cache.

That is working as expected for browser cache which isn’t same as CDN cache. For CDN cached objects, inspect the response header’s age header to determine if cached asset is coming from Cloudflare CDN cache.

example

image

Thanks again for the reply. I think we still have a misunderstanding.

First of all after 12 hours I am still getting a response back in Chrome that shows STALE for a resource with max-age of 20 seconds and an age of 42300. That’s the behavior I’m wanting to prevent and perhaps it just isn’t possible. Ironically I got into this rabbit hole because I was trying to implement stale-while-revalidate and couldn’t get it working.

I don’t want STALE content unless I’m setting stale-while-revalidate or stale-if-error.

This is all somewhat of an academic exercise at this point because if my server is up then I’ll get the refreshed content - and if my server is down for any length of time I have bigger problems!

Maybe the answer to my original question is simply:

‘If the origin cannot be contacted then regardless of Edge TTL rules or cache-control your content will be served as STALE indefinitely until Cloudflare purges the cache during routine maintenance.’

Again I am not setting any ‘Edge Cache TTL’ page rules.

Simon on the Cloudflare Team said:

Cloudflare will respect your origin expires / cache control headers to calculate the Edge Cache TTL (providing you don’t override it with a page rule setting an explicit Edge Cache TTL).

That’s exactly what I’m seeing. It’s simply not true that the minimum TTL for a Pro plan is 1 hour - that is ONLY a limitation of the dropdown.

I what you’re insisting is actually true then please explain how if I keep refreshing my page on different browsers and different devices that a breakpoint in my code is hit only one time in every 20 second window (and not 1 hour). That means the edge cache is expiring after 20 seconds. The limitations across plans is quite frankly just a marketing ploy.

I feel like this is possibly an Argo tunnel issue.

I had taken my local server down and it still kept coming as STALE.

I now took the Argo tunnel down and i get Error 1033: Argo Tunnel error.

stale-while-revalidate works on Cloudflare just not strictly to RFC specs. The specs say server MAY serve a stale response on revalidations but it doesn’t say must on every request. What Cloudflare does for stale-while-revalidate is it will only serve a stale asset on revalidation when there are concurrent requests to the same asset and not for non-concurrent requests. So what happens if you as a single request user do a non-concurrent request for an asset that revalidates, you will not get that stale response but either a expired/revalidated one as Cloudflare only will serve that stale asset if there are concurrent requests for that asset.

So if stale-while-revalidate is what you desire, and you can live with stale assets being served only for concurrent requests to that asset, then just set stale-while-revalidate header and be done with it :slight_smile: For me that is good enough as you only really need to do that when there’s alot of traffic which maybe of high concurrency in nature. Every other single non-concurrent request can get a revalidated asset.

Yes Cloudflare may expire assets sooner than min Edge Cache TTL if the asset is infrequently accessed from my understanding but what that threshold or criteria is I don’t know. Just be sure you’re testing properly with right request headers (via command line or browser/online tools) i.e. not no-cache etc and testing from same CF datacenter region as CF CDN cache is per datacenter.

Maybe I am still new to Argo Tunnels only started playing with it for my Wordpress blog and write up at https://blog.centminmod.com/2021/02/09/2250/how-to-setup-cloudflare-argo-tunnel-on-centos-7/.

Can you elaborate on how you are testing for STALE response so others here can try to replicate it.

I added a column to my Chrome dev tools - so I can see the cf-cache-status very easily each time I refresh the page. It’s very useful :slight_smile:

In this example the header was cache-control: public,max-age=20,stale-while-revalidate=60,max-stale=60,stale-if-error=60.

It was making an attempt to contact my origin every time - I got the following in my console:

ERRO[144159] HTTP request error error="Error proxying request to origin: dial tcp 127.0.0.1:5000: connectex: No connection could be made because the target machine actively refused it."

I’ll maybe try to test the same scenarios without Argo tunnel. I think maybe their ‘server down’ logic for Argo is tripping up the caching logic. Maybe it’s even by design!

I had actually already ‘invented’ a stale-if-error mechanism myself in a worker (refreshing in the background) but then I discovered that there was already a header supposed to do that.

At the end of the day it’s impossible to test stale-while-revalidate if I get a STALE response after 12 hours without even trying!

2 Likes

That might be part of the problem as per Understanding Cloudflare's CDN – Cloudflare Help Center a STALE response happens when

The resource was served from cache but is expired. Cloudflare couldn’t contact the origin to retrieve the updated resource.

So if your Argo Tunnel is down or misconfigured on origin server side, it could mean Cloudflare isn’t able to make contact with Argo Tunnel and give you a STALE response ?

also from Understanding Origin Cache-Control – Cloudflare Help Center for your stale-if-error at least

The stale-if-error directive is ignored if Always Online is enabled or if an explicit in-protocol directive (e.g., by a *no-store *or no-cache cache directive, a must-revalidate cache-response-directive, or an applicable s-maxage or proxy-revalidate cache-response-directive) is passed.

and

must-revalidate The must-revalidate response directive indicates that once it has become stale, a cache (client or proxy) must not use the response to satisfy subsequent requests without successful validation on the origin server.

maybe try just cache-control: public,max-age=20,stale-while-revalidate=60 and ensure a page rule for origin cache control is enabled

That’s the coolest thing I’ve seen in quite a while. Now I’m mad at @sandro because Firefox doesn’t let me do this. (sorry for the interruption)

5 Likes

I’ve done a few more tests and they’ll have to be my last for this round!

First of all when I go directly to my server and not through Argo tunnel it takes 15 seconds rather than 2 to determine that the server is down but it is still willing to return a STALE response - even well after max-age and stale-while-revalidate times have elapsed.

However after a while (maybe an hour) it did eventually decide to give me an error back - which is much less than the 12+ hours I saw with Argo. Still not sure I proved anything with that informal test.

This is what I just tested now regarding revalidation:

cache-control: public,max-age=20,stale-while-revalidate=60,max-stale=60,must-revalidate,stale-if-error=60

must-revalidate is preventing ANY STALE responses being sent even within the 60 second window. I’m beginning to wonder if stale-while-revalidate is a Business only feature or something else like that?

I’ll have to revisit this again another time.

Just checked my Cache Analytic stats and can confirm stale-while-revalidate works as I have CF Worker doing guest full page HTML page caching on my blogs and forums and it uses stale-while-revalidate header set to 60 seconds and can see STALE requests served up mainly to my robots.txt file as I guess search bots do have concurrent request traffic patterns

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.