Getting Cache MISS after 24 hours even though headers are set to cache longer

nicholas.waton · April 14, 2021, 7:05pm

I am having a performance with caching.
I am using WordPress with the ShortPixel adaptive images plugin to optimize my images on my website (www.nicholaswaton.com). I have set up a subdomain to serve the images from the Shortpixel optimizer.
I have been in touch with ShortPixel and they’ve set up their headers to make sure images are cached in CDNs for 30 days.

But I have set up a monitoring test on GTMetrix, and this automated test shows that I get cache MISS on any images served by ShortPixel AI.
here’s a test from yesterday: https://gtmetrix.com/reports/www.nicholaswaton.com/uHYcEWMB/
and today: https://gtmetrix.com/reports/www.nicholaswaton.com/R5DkHHqP/
but if I re-test a few minutes later, I get much higher speed and the images as Cache HIT
https://gtmetrix.com/reports/www.nicholaswaton.com/HHb5OzLh/

So it seems the images expire every day instead of being cached by CF

Just in case, I have set up also a page rule in CF to cache everything and to set the Edge TTL to one month. But it seems all these rules are not taken into account. These images never change.

I have been in touch with Shortpixel and they suggested for me to get in touch with CF.

What am I doing wrong? How can I make sure to get Cache HIT most of the time?

sdayman · April 14, 2021, 7:09pm

Edge Cache TTL is not a guarantee it will stay there for a month. Edge Cache TTL is more like a maximum amount of time it’s safe to cache a resource. Low-usage files get evicted from the cache early to make room for high-use resources.

nicholas.waton · April 14, 2021, 7:40pm

No I didn’t. Where do you see this? When I go to my website it shows I have a valid certificate from CF.

nicholas.waton · April 14, 2021, 7:41pm

Everything on www.nicholaswaton.com is getting a cache HIT, only things on img.nicholaswaton.com is getting a cache MISS even though the headers are saying to cache it longer.

sdayman · April 14, 2021, 7:54pm

First look got me this:

oneandonlyjason · April 14, 2021, 7:55pm

Im getting a Cache HIT right now, but that doesnt mean that it is like this forever. Like already mentioned by @sdayman the max Cache Age and Cloudflare Edge Cache TTL only shows Cloudflare how long it is allowed to cache the File, but Cloudflare itself decides how long they will cache it in Reality. When a File is only Requested once every 24 Hours and the Colo that answered the Request has a lot of Traffic the File will get purged from Cloudflare Cache to make Space for Files that are requested more often

nicholas.waton · April 22, 2021, 5:16pm

I am having a performance with caching.
I am using WordPress with the ShortPixel adaptive images plugin to optimize my images on my website (www.nicholaswaton.com). I have set up a subdomain to serve the images from the Shortpixel optimizer.
I have been in touch with ShortPixel and they’ve set up their headers to make sure images are cached in CDNs for 30 days.

But I have set up a monitoring test on GTMetrix, and this automated test shows that I get cache MISS on any images served by ShortPixel AI.
here’s a test from yesterday: Archived Performance Report for: https://www.nicholaswaton.com/ | GTmetrix
and today: Archived Performance Report for: https://www.nicholaswaton.com/ | GTmetrix
but if I re-test a few minutes later, I get much higher speed and the images as Cache HIT
Latest Performance Report for: https://www.nicholaswaton.com/ | GTmetrix

So it seems the images expire every day instead of being cached by CF

Just in case, I have set up also a page rule in CF to cache everything and to set the Edge TTL to one month. But it seems all these rules are not taken into account. These images never change.

I have been in touch with Shortpixel and they suggested for me to get in touch with CF.

What am I doing wrong? How can I make sure to get Cache HIT most of the time?

I ended up deactivating Cloudflare to serve these images as the ShortPixel CDN is serving them much faster. You can see the improvement of performance on this graph. I removed the Cloudflare configured subdomain around April 15th.

cbrandt · April 22, 2021, 6:28pm

Hi,

Only a guess, but Its my impression that Cloudflare is not playing well with the fact that your images have a Content-type: image/webp header while their extension is the original image format (.jpg, for instance).:

https://img.nicholaswaton.com/spai/w_651+q_glossy+ret_img+to_webp/https://www.nicholaswaton.com/wp-content/uploads/2021/01/dscf5673-e1612294240519-uai-1813x1360.jpg

Check if your image optimization plugin has an option to change the filename extension, perhaps by appending .webp to the image URL.

Cloudflare only caches based on file extension and doesn’t cache by MIME type

quoted from:
https://support.cloudflare.com/hc/en-us/articles/200172516

oneandonlyjason · April 23, 2021, 1:23am

That only applies for the question if cloudflare should Cache a File. He has a Pagerule in place to Cache Anything so the Extension is no Problem.
It could be a .bin and Cloudflare would still Cache it with this Rule in Place. And after a Visit i got Hit for images before he deactivated the Proxy for this Subdomain. Cloudflare simply seems to delete the Files out of the Cache pretty quickly. Probably to less Traffic so the Colo deleted the Cache after a few Hours

nicholas.waton · April 27, 2021, 3:39pm

@morehelp

1blas · April 27, 2021, 5:41pm

I ran a cURL from the SJC colo to this URL (found in the GTMetrix results you shared), not only did I get a MISS, but I also see that the image was modified recently < last-modified: Fri, 23 Apr 2021 16:54:16 GMT

> GET /spai/w_651+q_glossy+ret_img+to_webp/https://www.nicholaswaton.com/wp-content/uploads/2021/01/dscf5357-uai-1820x1365.jpg HTTP/2
> Host: img.nicholaswaton.com
> Accept: */*
> User-Agent: cf-crossbow/660-fa9cce4
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [238 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [238 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
} [5 bytes data]
< HTTP/2 200
< date: Tue, 27 Apr 2021 17:36:02 GMT
< content-type: image/webp
< content-length: 68910
< set-cookie: __cfduid=db9d7dae20b1fc79f4a3d02d463f8c8a01619544961; expires=Thu, 27-May-21 17:36:01 GMT; path=/; domain=.nicholaswaton.com; HttpOnly; SameSite=Lax; Secure
< link: <https://www.nicholaswaton.com/wp-content/uploads/2021/01/dscf5357-uai-1820x1365.jpg>; rel="canonical"
< etag: 754e984d801ab2fe2ca845684f39c2b4
< x-tag: Domain:nicholaswaton.com
< pragma: cache
< vary: Accept-Encoding
< expires: Sat, 23 Apr 2022 15:54:16 GMT
< cache-control: max-age=31184295, s-maxage=2240295
< last-modified: Fri, 23 Apr 2021 16:54:16 GMT
< cf-cache-status: MISS

If we look at that URL from the GTMetrix results, I can see a different last-modified header:

|accept-ranges|bytes|
| --- | --- |
|alt-svc|h3-27=":443"; ma=86400, h3-28=":443"; ma=86400, h3-29=":443"; ma=86400|
|cache-control|max-age=31536000, s-maxage=783807|
|cf-cache-status|MISS|
|cf-ray|63f72b4fbe430252-SJC|
|cf-request-id|096e5965db0000025234225000000001|
|content-length|68910|
|content-type|image/webp|
|date|Tue, 13 Apr 2021 19:40:59 GMT|
|etag|754e984d801ab2fe2ca845684f39c2b4|
|expect-ct|max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"|
|expires|Wed, 23 Mar 2022 21:24:25 GMT|
|last-modified|Tue, 23 Mar 2021 21:24:25 GMT|

If a resource is modified, then you would in theory see a MISS as the Cloudflare cache server would need to then reach back out to your origin server for a fresh copy of the file to cache when requested.

Is this expected behavior? Do you have something in place that would be modifying the resources, or are you manually changing them?

eva2000 · April 27, 2021, 7:51pm

sounds like their use of ShortPixel Adaptive Images – WebP, AVIF, CDN, Image Optimization – WordPress plugin | WordPress.org

Images are processed on the fly and served from our CDN, in the next-gen WebP format, if the browser supports it.

@nicholas.waton ShortPixel Adaptive Images plugin isn’t really needed if you’re using Cloudflare in front especially with Pro and higher plans and Polish webP.

michael · April 27, 2021, 8:13pm

I think the OPs question is why the assets respond HIT today, but MISS tomorrow, when they have an Edge TTL of a month.

The answer is most likely “because they are not popular enough”. Cloudflare will not cache an infinite amount of content, and the least popular content will be evicted first. So if you have a low traffic site, your content will not be cached for as long as a month.

There is an Images Beta starting, which may avoid this.

ncano · April 28, 2021, 9:35am

It seems that the edge cache TTL rule is no longer in place.

Could you please add it back and let me know which colo your affected requests hit? You can determine it from cf-ray header value.

nicholas.waton · April 28, 2021, 2:30pm

Ok I’ve reactivated img.nicholaswaton.com on my website.So if you go to https://www.nicholaswaton.com/
The image are served through Cloudflare again.
The images are served via img.nicholaswaton.com which sits in front of ShortPixel AI, which serves the images from my website.

I’ve also put back the edge cache TTL rule.

nicholas.waton · April 28, 2021, 9:48pm

@eva2000 well so far ShortPixel AI is giving much better performance and at a much cheaper cost than Cloudflare. I don’t really have the traffic to justify paying for Cloudflare Pro ($20/month). With ShortPixel AI you only pay for the number of images you optimize. 30.000 for $20. So I’ve been using ShortPixel AI for a few month, and have had just a one-time fee. Since January I’ve only used 13,000 credits.

Maybe Cloudflare is just not justified for my particular case. It is still surprising that Cloudflare would delete just the image assets every 24 hours. The rest is kept. At least it should be made a little bit more clear when you set up the configuration.

eva2000 · April 28, 2021, 10:08pm

I use ShortPixel but not the AI plugin as I have a ShortPixel lifetime account but AI plugin requires jQuery but my Wordpress site doesn’t use jQuery (better page speed)

However, the $20 CF Pro plan does more than CDN serving. You get access to CF WAF and more Firewall rule/page rule quotas for security. And for page speed performance you gain Cloudflare TCP Turbo and Enhanced HTTP/2 Prioritization which may do more for page speed performance than what Shortpixel AI can do alone - given the particular current Google Core Web Vital metrics being focussed on.

Regarding CF Enhanced HTTP/2 Prioritization

This prioritization scheme allows sending the render-blocking content serially, followed by the visible images in parallel and then the rest of the page content with some level of sharing to balance application and content loading. The “* If Detectable” caveat is that not all browsers differentiate between the different types of stylesheets and scripts but it will still be significantly faster in all cases. 50% faster by default, particularly for Edge and Safari visitors is not unusual:

If you were on CF Pro and higher, you could utilise CF Cache Analytics to track over time a specific URL’s cache status so you could in theory see what goes from cache hit to miss and how many requests are made. Another advantage of paid Pro and higher plans

example tracking one of my images over last week

serving webP extension images via ShortPixel plugin conversion and picture srcset HTML code to workaround CF’s lack of HTTP Vary header support (there’s no need for ShortPixel AI in this case)

nicholas.waton · April 29, 2021, 3:14am

Thanks for all these insights. Very interesting.

lcrowter · April 30, 2021, 8:27am

I think the answers throughout the thread and the great reply by @eva2000 address this specifically when Cloudflare is being used with ShortPixel, so I’m going to close this off for now.

lcrowter · April 30, 2021, 8:27am