"Cache Everything" for existing exe file will cause 404 errors intermittently

Hi,

I notice that when I set up to cache everything for large exe files(about 10MB). Then I will get 404 errors intermittently when downloading these files, though they are existing.

For example, the exe file https://www.datanumen.com/downloads/files/doeu.exe will cause 404 if “Cache Everything” is enabled.

However, if I change the cache policy to “ByPass”, then the exe file can be downloaded without problems.

The server will return

cache-control: no-cache, must-revalidate, max-age=0

for 404 status, which means the next time when I access the URL, Cloudflare will revalidate the file from the origin.

I have also tried to add the following line in .htaccess to make sure the edge only cache the 404 status for 3 minutes:

Header set CDN-Cache-Control "max-age=180, must-revalidate" "expr=%{REQUEST_STATUS} == 404"

However, after 3 minutes, when I re-download the EXE file, I still get the 404 error.

How to solve this issue?

I find this post 404 cache not clearing - #2 by jahed which also try many methods to solve the issue, including add no-cache header but not working.

Could you grab and share a screenshot of this error?
Is it coming from your origin host / server or rather somehow from Cloudflare for the requested resource?

Could you check if there is something else preventing to load the URL/resource, like AntiVirus program, Web browser security extension, etc.?

Could you share an example URL with us here?

May I ask have you selected the option Origin Cache Control toggle “On” using a Page Rule alongside the Cache Level: Cache Everything option?

Hi, @fritex

Thank you very much.

Now I am checking the system access log to see why 404 will occurs for existing file.

Anyway, you can see a sample URL at https://www.datanumen.com/downloads/files/doeu.exe , I have just enabled the “Origin Cache Control” for the file. But it seems the following cache-control rule still not work:

cache-control: no-cache, must-revalidate, max-age=0

BTW: As far as I can remember, Cloudflare will always respect the origin cache control, see https://developers.cloudflare.com/cache/about/default-cache-behavior, so why bother to enable that option “Origin Cache Control” again?

I see you are using APO and Tiered Caching, right?

cf-apo-via: tcache
cf-cache-status: HIT

Have you tried to manually purge the cache via the Cloudflare dashboard for a specific URL (https://www.datanumen.com/downloads/files/doeu.exe)?, Does this work?

Also, maybe to try purging both at APO plugin interface and also at Cloudflare dashboard - Purge Everything?

cache-control: no-cache, must-revalidate, max-age=0
cdn-cache-control: max-age=180, must-revalidate

Maybe @yevgen could take a look at this one.

@fritex

Yes, we are using APO plugin for WordPress. Also we are using Argo Tiered Cache as well.

And yes, if we purge the URL manually, then the issue will be solved. However, after sometimes, one or several files will have such a problem again. So, what we need is to solve the issue for ever, not just for now.

1 Like

Hi,

By default, Cloudflare will cache 404 for 3 mins. However, now I setup a rule:

  1. Cache Everything
  2. Set Edge TTL to 14 days.
  3. Origin Cache Control: On

Then Cloudflare will cache everything, including 404 for 14 days in the edge server. To keep it continue caching 404 for 3 mins, while cache others for 14 days, I addded the following cache control headers:

cache-control: no-cache, must-revalidate, max-age=0
cdn-cache-control: max-age=180, must-revalidate

But this does not work. I make the following test:

  1. Try to access an unexisting file https://www.datanumen.com/downloads/files/dorcr1.exe and let Cloudflare cache the 404.
  2. After 3 mins, upload the file dorcr1.exe.
  3. Reacces the file. Since the file now exists, and it has passed the max-age(3min) for cdn-cache-control, so it should not return 404.
    But the test results shows that URL will continue return 404, until I purge the cache for that URL manually in Cloudflare.

So it seems that the cache control headers does not work on Cloudflare at all? Even though I have enabled “Origin Cache Control: On”?

Are you sure your origin headers are set like this? I checked and I see:

< cache-control: max-age=2592000, public
< expires: Sun, 23 Jan 2022 10:22:42 GMT

Try using cURL to check:

curl -svo /dev/null https://www.datanumen.com/downloads/files/dorcr1.exe --connect-to ::1.2.3.4

Replace 1.2.3.4 with your origin IP.

1 Like

@simon I believe it is related to a recent topic from here (just in case if needed):

Thanks @CroatiaHR - I have merged these topics to keep things in one place.

@ccw I checked your origin again for a HTTP 404 response and I do see the cache headers you mention.

Is the Wordpress APO plugin installed? Can you confirm the version of the plugin and of Wordpress? Do you have any other caching plugins installed?

3 Likes

@simon,

Is the Wordpress APO plugin installed?
A: Yes.

Can you confirm the version of the plugin and of Wordpress?
A: Plugin version: 4.7.0. WordPress version: 5.8.2.

Do you have any other caching plugins installed?
A: We have installed WP Rocket 3.10.5.1, but does not enable its Addon for Cloudflare.

May I suggest below to posts related to APO and WP Rocket:

@CroatiaHR

Thank you for your suggestions. However, these posts do not provide a solution for 404 status code.

With the help of Cloudflare support, I am finally able to solve the issue.

In https://developers.cloudflare.com/cache/about/cdn-cache-control#edge-cache-ttl-page-rule, it said " The Edge Cache TTL page rule overrides the amount of time an asset is cached on the edge (Cloudflare data centers). This page rule overrides directives in Cloudflare-CDN-Cache-Control/CDN-Cache-Control which manage how long an asset is cached on the edge."

So, to set different edge TTL for normal objects and 404, we should do as follows:

  1. Remove all Edge Cache TTL settings.

  2. In .htaccess, add the following line:

# Set edge TTL to 14 days for normal objects
Header set CDN-Cache-Control "max-age=1209600"
Header set Cloudflare-CDN-Cache-Control "max-age=1209600"

# Set edge to not cache 404
Header set CDN-Cache-Control "no-cache, must-revalidate, max-age=0" "expr=%{REQUEST_STATUS} == 404"
Header set Cloudflare-CDN-Cache-Control "no-cache, must-revalidate, max-age=0" "expr=%{REQUEST_STATUS} == 404"

After that, the edge will cache normal objects for 14 days and not cache 404, working as expected.

Thanks to all of your helps.

3 Likes

Hi @ccw - glad you got this worked out and thanks for sharing the solution here - really helpful.