Cloudflare is caching empty files after changing IP in DNS

Website, Application, Performance
1

On May 9, 2023, I changed the IP that my Cloudflare site was pointing to. It has been more than a week since then, so DNS changes should have definitely propagated.

Since then, Cloudflare randomly starts to cache empty files (in the request header, these have a “content-length” of 0). Any time Cloudflare updates its cache for a file, the cached file is empty. All files that still work that existed on the site before the IP change have a “last-modified” date of before when the IP was changed, namely May 5th. All of the files that still work have a cf-cache-status of EXPIRED.

To temporarily fix the issue, you can change the query string for the file, and it will retrieve a previous version of it that Cloudflare has in its cache, which is always expired.

Here are the request headers for a working image (taken on May 17):

HTTP/2 200 OK
date: Wed, 17 May 2023 16:06:18 GMT
content-type: image/png
content-length: 1625
x-powered-by: Express
cache-control: public, max-age=14400
last-modified: Fri, 05 May 2023 21:53:32 GMT
etag: W/“659-187ede7eca8”
cf-cache-status: EXPIRED
accept-ranges: bytes
report-to: {“endpoints”:[{“url”:“https://a.nel.cloudflare.com/report/v3?s=6%2FqVtXJwbqqS%2FVA%2BF7CraYLHqmD6cph2oyuzEMqnF5LkV71CGB3IxCzXHYBLNKSXWfj8s1R9eT9swZ800f0PiyzWBMFipat6cU3UEm3ZC4Pv7efnzbNlApF6gA%3D%3D”}],“group”:“cf-nel”,“max_age”:604800}
nel: {“success_fraction”:0,“report_to”:“cf-nel”,“max_age”:604800}
vary: Accept-Encoding
server: cloudflare
cf-ray: 7c8d1b602c4d1725-IAD
X-Firefox-Spdy: h2

Reloading the page still keeps the cf-cache-status EXPIRED. Note that the last-modified was before I changed where the IP for where the DNS was pointing.

For a broken image, the headers look as follows:

HTTP/2 200 OK
date: Wed, 17 May 2023 16:13:45 GMT
content-length: 0
last-modified: Wed, 17 May 2023 15:02:08 GMT
cache-control: max-age=14400
cf-cache-status: HIT
age: 20
accept-ranges: bytes
report-to: {“endpoints”:[{“url”:“https://a.nel.cloudflare.com/report/v3?s=sNaMKdT73MDaZ9BUtoWzIscVWM6rpEZmyQwznYoH1gPstfGQQQTpkARd1dmImmDpu0lpE%2FrIBNMUVAVbRGQVhEjlxR5Km5ZuB%2B%2BTxXvA9vsZ8QENH901a4Rn4%2BVedD39”}],“group”:“cf-nel”,“max_age”:604800}
nel: {“success_fraction”:0,“report_to”:“cf-nel”,“max_age”:604800}
vary: Accept-Encoding
server: cloudflare
cf-ray: 7c8d2646fa355a0a-IAD
alt-svc: h3=“:443”; ma=86400, h3-29=“:443”; ma=86400
X-Firefox-Spdy: h2

Purging Cloudflare cache temporarily fixes this issue, until Cloudflare starts caching these empty files again. Perhaps Cloudflare is trying to query the old IP (which is no longer online) for the static content to cache? All files that are loading that existed before the DNS change has a “last-modified” of before the DNS settings were changed. All files that existed prior to the DNS change that have a later “last-modified” date will return an empty file.

I’ve also tried creating a page rule to set the Edge TTL to be very short (2-4 hours), but that did not fix anything.

How do I fix this and have Cloudflare cache the files normally? Any help on this would be greatly appreciated!

4

Is this only happening with image files (as in your first example)?

If so, do you make use of any image transformation feature, such as Cloudflare Images, Image Resizing, or Polish?

Are affected URLs processed by Workers or Transform Rules?

Do you mind to share a couple of affected URLs?

5

Thank you for your quick reply!

This is happening with all static content that can be cached by Cloudflare by default, including .css files and .js files, rendering the site unusable at times.

I do not have any image transformations enabled, and I do not use Workers or Transform Rules.

I don’t have an example anymore of a URL where the content-length is 0, as a URL that was previously broken has seemingly magically fixed. Since then I have been attaching the UNIX time (rounded down to every 30 minutes) to the query string to force the pages to retrieve the files from the origin server.

I do, however, have the .HAR file for when I loaded a broken URL with an empty file, available here: https://textbin.net/y7wfnyfqc7

Here is a link to an image that loads, but has a cf-cache-status of EXPIRED along with the out-of-date last-modified: https://battlefun.org/static/images/menu-ability1.png?ver=0.0.8a-1

Let me know if you need any more information.

6

Another thing I forgot to mention is that before the IP change, the old server was encrypted end-to-end, with a self-signed certificate on the server. The new server does not have a certificate, so Cloudflare is using Flexible SSL/TLS encryption now.

7

You can always use a Cloudflare Origin Certificate to secure your server if you do not have access to other certificates.
See Origin CA certificates · Cloudflare SSL/TLS docs for more info.

2 Likes
8

Thank you for the link, I was planning to do this soon. But this does not address my issue.

9

Could the issue be the result of Etag headers and the way Cloudflare handles them?

When using weak ETag headers, disable Email Obfuscation and Automatic HTTPS Rewrites to ensure Cloudflare does not remove the ETag headers set by your origin web server.
Source: Using ETag Headers with Cloudflare · Cloudflare Cache (CDN) docs

1 Like
10

Thanks for the reply!

I’ve disabled Automatic HTTPS Rewrites and purged the cache, but the issue is still happening:
https://battlefun.org/static/images/menu-ability1.png?ver=0.0.8a-1

Same thing, cf-cache-status is still EXPIRED and the last-modified is still May 5th.

11

Is it possible for you to test disabling Etag generation at the origin, even if only for a couple of URLs, see if that changes results?

12

Thanks for the reply.

I’ve added the line

etag off;

to my nginx configuration for downsurge.pro, another URL that is pointing to the same new IP that is experiencing the same issue. Disabling automatic HTTPS rewrites, clearing the cache and reloading nginx, the issue remains:

https://downsurge.pro/static/images/menu-ability1.png

The cf-cache-status is still EXPIRED.

13

I think the issue is not what I initially hypothesized – I have another URL, supremtestserver1.online, which I added to Cloudflare very recently and pointed directly to the new IP, and it is also experiencing the same issue:

https://supremtestserver1.online/static/images/menu-ability1.png

This means that it’s probably not Cloudflare trying to query the old IP for the cache, and is perhaps a issue with the origin. However, I have no idea as to what this issue may be or where to even start. Any pointers?

14

I still see an Etag header for that URL, even with a MISS (with a query string)

image

15

Ah, the etags were being generated by Express and not by nginx. I’ve disabled them now and purged the cache on Cloudflare, but the issue continues:
image

Also, I figured out why the last-modified is May 5th 2023. This is when the server was first created and the files were uploaded! So that has probably nothing to do with the issue.

16

Right, but it still points out that your whole site is insecure. For that reason, the statement was spot on and you should definitely switch ASAP to Full Strict, not only because your current legacy mode is known for all sorts of issues, such as the one here.

1 Like
17

Thanks for the reply. I’ve installed a Cloudflare Origin Certificate for downsurge.pro, and switched that domain to Full (Strict), and then purged the cache. The issue is still occuring:
https://downsurge.pro/static/images/menu-ability1.png

image

Hard refreshing doesn’t do anything.

18

Not the same issue, though. You initially said it was returning Content-Length: 0, which isn’t happening any more.

As for the Expired cache status, that’s expected when you have Cache-Control: public, max-age=0. If you cannot change that at the origin, you should try a Cache Rule with instructions to override Browser Cache TTL.

19

Thanks for everyone’s help! I think both issues have been resolved by adding an origin certificate as Laudian and sandro suggested and switching to Full (Strict), as well as changing the max-age header as cbrandt suggested.

1 Like
20

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.