Resources expire from Cloudflare cache more frequently than expected

[emiliano.maina]

Hello,

I am in the process to understand how to improve Cloudflare cache ratio changing what is required on my blog server side. At the moment I cannot guess anything more and anyway CF hit ratio is quite low.

I made a very simple test using curl to get response header of one static resource from the same IP every 15 seconds for about one hour monitoring the age field using this script:

for i in {1..240}
do
    echo $(date -u) >> log/cloudflare_test.log
    curl -I https://lamiacasaelettrica.com/wp-content/uploads/2021/08/cropped-la-mia-casa-elettrica-logo-192x192.png | grep age: >> log/cloudflare_test.log
    sleep 15
done

One sample response header is the following

HTTP/2 200
date: Tue, 17 Aug 2021 12:04:42 GMT
content-type: image/png
content-length: 6490
cache-control: public, max-age=31536000
cf-bgj: imgq:85,h2pri
cf-polished: origSize=8395
etag: "20cb-5c892f0b41229"
expires: Wed, 17 Aug 2022 11:41:26 GMT
last-modified: Mon, 02 Aug 2021 12:45:12 GMT
pragma: public
referrer-policy: no-referrer-when-downgrade
strict-transport-security: max-age=31536000; includeSubDomains
vary: Accept
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-powered-by: W3 Total Cache/2.1.6
cf-cache-status: HIT
age: 1395
accept-ranges: bytes
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=C2v86UKTSDfDAvoK%2FL9TTlOPQagL7IPOZAVlReUHsQ4kjQyLAy%2BtMeW%2FJ%2FUKOHzzozYD%2Bgo9UG2WPeQ%2F13s%2FmEouDSTid6yvyDHduN1PMedOLmq5Y4a7G02LnhLvWaxAIl645MY1bnw%3D"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
server: cloudflare
cf-ray: 6802c437399c0e2a-MXP
alt-svc: h3-27=":443"; ma=86400, h3-28=":443"; ma=86400, h3-29=":443"; ma=86400, h3=":443"; ma=86400

As you can see the resource is “old” and not to be expired as per my server response header:

date: Tue, 17 Aug 2021 12:04:42 GMT
...
expires: Wed, 17 Aug 2022 11:41:26 GMT
last-modified: Mon, 02 Aug 2021 12:45:12 GMT

The resulting graph of age field is the following:

For completeness below you can find also W3 Total Cache WordrPress plugin Cloudflare extension configuration:

The very simple question is: why CF cache is expiring so frequently?

In the sample test maximum age was 2655 seconds (44 minutes).

As per all the configurations it should be really longer time.

Any answer or suggestion would be beneficial. Thanks in advance.

[erictung]

Does your website has a high number of visitor traffic? Cloudflare usually expires unpopular assets earlier than the TTL that you specified to make more room for more popular assets to be cached.

[emiliano.maina]

These are CF analytics:

For my test I selected an image that is requested in every page.

Where is the documentation of this algorithm “Cloudflare usually expires unpopular assets earlier than the TTL that you specified to make more room for more popular assets to be cached” ?

[erictung]

@emiliano.maina

Where is the documentation of this algorithm “Cloudflare usually expires unpopular assets earlier than the TTL that you specified to make more room for more popular assets to be cached” ?

Although this info is not officially disclosed by Cloudflare itself (at least I can’t find it in any Cloudflare documentation yet), but I believe other @MVP can provide more evidence about this. And yes, this has been discussed several times previously and the answer is similar to what I told you in my previous reply.

[sdayman]

I’m pretty sure the algorithm is extremely fluid, so you’ll never be able to pin down a pattern. Especially since everybody’s traffic is different, as is edge node traffic. Here are a couple of somewhat recent blog posts on how they run the cache:

Why We Started Putting Unpopular Assets in Memory

Third Time’s the Cache, No More

[emiliano.maina]

Thanks for the answer. I like the article posted very much.

If I understand well CF algorithm, sites with greater traffic would have greater hit rate because they have more popular assets.

But the point is that the monthly CF service fee is not fluid and not depending on traffic

So basically small traffic sites owner are paying to provide a better service for big traffic sites. Not so nice if I am right.

So, considering the technical evidence provided above that on my Wordpress server side I cannot do more to “tell” to CF to cache resources, how CF could reach grater hit rate? Let’s say al least > 50%.

[eva2000]

@emiliano.maina

The very simple question is: why CF cache is expiring so frequently?

Besides grep filtering age response header, you might want to grep filter on cf-ray header too as CF CDN cache is per datacenter, so you want to be sure that CF is serving all you test requests to same Milan MXP CF datacenter. If some come from other CF datacenters, then if could be cache misses too.

But yes, CF CDN will expire infrequently requested assets regardless of cache TTL set.

Also make sure if you are using any Wordpress plugins with tie into CF Cache via API key, that they aren’t purging CF Cache with certain Wordpress actions i.e. updating/posting new blog posts, update WP theme etc.

@emiliano.maina

So, considering the technical evidence provided above that on my Wordpress server side I cannot do more to “tell” to CF to cache resources, how CF could reach grater hit rate? Let’s say al least > 50%.

Make sure to filter CF Cache analytics for known cacheable requests first to filter out requests which won’t be cached i.e. non-200 HTTP Status requests like 403, 404 or /cdn-cgi requests and then look at your cache hit rate averages.

Example, here’s one of my site’s CF Cache analytics unfiltered at 52% cache hit rate. And 2nd screenshot is filtered only for 200 HTTP status and minus cdn-cgi and any url which I do not expect to cached (I told it not to cache) = 84%

[emiliano.maina]

@eva2000

Besides grep filtering age response header, you might want to grep filter on cf-ray header too as CF CDN cache is per datacenter, so you want to be sure that CF is serving all you test requests to same Milan MXP CF datacenter. If some come from other CF datacenters, then if could be cache misses too.

Done it. As expected using always Milan MXP (we have onyl 2 CF node in Italy).

@eva2000

Make sure to filter CF Cache analytics for known cacheable requests first to filter out requests which won’t be cached i.e. non-200 HTTP Status requests like 403, 404 or /cdn-cgi requests and then look at your cache hit rate averages.

Done it.

Overall:

Restricting to 200 and onyl calls from Italy (real users):

I am quite far from your hit rates

@eva2000

Also make sure if you are using any Wordpress plugins with tie into CF Cache via API key, that they aren’t purging CF Cache with certain Wordpress actions i.e. updating/posting new blog posts, update WP theme etc.

I think this a good point: are there way to investigate this on CF server side APIs calls?
Unfortunately many WP plugins are “let me say” black-boxed, especailly on how they could call CF APIs.

My only candidate is W3 Total Cache (that could be “pushed” from other plugins).