"Cache Everything" for existing exe file will cause 404 errors intermittently

[ccw]

Hi,

I notice that when I set up to cache everything for large exe files(about 10MB). Then I will get 404 errors intermittently when downloading these files, though they are existing.

For example, the exe file https://www.datanumen.com/downloads/files/doeu.exe will cause 404 if “Cache Everything” is enabled.

However, if I change the cache policy to “ByPass”, then the exe file can be downloaded without problems.

The server will return

cache-control: no-cache, must-revalidate, max-age=0

for 404 status, which means the next time when I access the URL, Cloudflare will revalidate the file from the origin.

I have also tried to add the following line in .htaccess to make sure the edge only cache the 404 status for 3 minutes:

Header set CDN-Cache-Control "max-age=180, must-revalidate" "expr=%{REQUEST_STATUS} == 404"

However, after 3 minutes, when I re-download the EXE file, I still get the 404 error.

How to solve this issue?

I find this post 404 cache not clearing - #2 by jahed which also try many methods to solve the issue, including add no-cache header but not working.

[fritex]

@ccw

404 errors

Could you grab and share a screenshot of this error?
Is it coming from your origin host / server or rather somehow from Cloudflare for the requested resource?

• https://support.cloudflare.com/hc/en-us/articles/115003014512-4xx-Client-Error

Could you check if there is something else preventing to load the URL/resource, like AntiVirus program, Web browser security extension, etc.?

Could you share an example URL with us here?

@ccw

tried to add the following line in .htaccess

May I ask have you selected the option Origin Cache Control toggle “On” using a Page Rule alongside the Cache Level: Cache Everything option?

[ccw]

Hi, @fritex

Thank you very much.

Now I am checking the system access log to see why 404 will occurs for existing file.

Anyway, you can see a sample URL at https://www.datanumen.com/downloads/files/doeu.exe , I have just enabled the “Origin Cache Control” for the file. But it seems the following cache-control rule still not work:

cache-control: no-cache, must-revalidate, max-age=0

BTW: As far as I can remember, Cloudflare will always respect the origin cache control, see Default Cache Behavior · Cloudflare Cache (CDN) docs, so why bother to enable that option “Origin Cache Control” again?

[fritex]

I see you are using APO and Tiered Caching, right?

cf-apo-via: tcache
cf-cache-status: HIT

• Verify APO works · Cloudflare Automatic Platform Optimization docs

Have you tried to manually purge the cache via the Cloudflare dashboard for a specific URL (https://www.datanumen.com/downloads/files/doeu.exe)?, Does this work?

Also, maybe to try purging both at APO plugin interface and also at Cloudflare dashboard - Purge Everything?

cache-control: no-cache, must-revalidate, max-age=0
cdn-cache-control: max-age=180, must-revalidate

Maybe @yevgen could take a look at this one.

[ccw]

@fritex

Yes, we are using APO plugin for WordPress. Also we are using Argo Tiered Cache as well.

And yes, if we purge the URL manually, then the issue will be solved. However, after sometimes, one or several files will have such a problem again. So, what we need is to solve the issue for ever, not just for now.

[ccw]

Hi,

By default, Cloudflare will cache 404 for 3 mins. However, now I setup a rule:

1. Cache Everything
2. Set Edge TTL to 14 days.
3. Origin Cache Control: On

Then Cloudflare will cache everything, including 404 for 14 days in the edge server. To keep it continue caching 404 for 3 mins, while cache others for 14 days, I addded the following cache control headers:

cache-control: no-cache, must-revalidate, max-age=0
cdn-cache-control: max-age=180, must-revalidate

But this does not work. I make the following test:

1. Try to access an unexisting file https://www.datanumen.com/downloads/files/dorcr1.exe and let Cloudflare cache the 404.
2. After 3 mins, upload the file dorcr1.exe.
3. Reacces the file. Since the file now exists, and it has passed the max-age(3min) for cdn-cache-control, so it should not return 404.
   But the test results shows that URL will continue return 404, until I purge the cache for that URL manually in Cloudflare.

So it seems that the cache control headers does not work on Cloudflare at all? Even though I have enabled “Origin Cache Control: On”?

[simon]

Are you sure your origin headers are set like this? I checked and I see:

< cache-control: max-age=2592000, public
< expires: Sun, 23 Jan 2022 10:22:42 GMT

Try using cURL to check:

curl -svo /dev/null https://www.datanumen.com/downloads/files/dorcr1.exe --connect-to ::1.2.3.4

Replace 1.2.3.4 with your origin IP.

[CroatiaHR]

@simon I believe it is related to a recent topic from here (just in case if needed):

"Cache Everything" for existing exe file will cause 404 errors intermittently

@fritex Yes, we are using APO plugin for WordPress. Also we are using Argo Tiered Cache as well. And yes, if we purge the URL manually, then the issue will be solved. However, after sometimes, one or several files will have such a problem again. So, what we need is to solve the issue for ever, not just for now.

[simon]

Thanks @CroatiaHR - I have merged these topics to keep things in one place.

@ccw I checked your origin again for a HTTP 404 response and I do see the cache headers you mention.

Is the Wordpress APO plugin installed? Can you confirm the version of the plugin and of Wordpress? Do you have any other caching plugins installed?

[ccw]

@simon,

Is the Wordpress APO plugin installed?
A: Yes.

Can you confirm the version of the plugin and of Wordpress?
A: Plugin version: 4.7.0. WordPress version: 5.8.2.

Do you have any other caching plugins installed?
A: We have installed WP Rocket 3.10.5.1, but does not enable its Addon for Cloudflare.

[CroatiaHR]

May I suggest below to posts related to APO and WP Rocket:

Cloudflare Plugin, APO and WP-Rocket

From this article: By default, it’s not necessary to press this button every time you clear WP Rocket’s cache. It will clear the Cloudflare cache, and it mostly needed for troubleshooting, in case you’ve made changes to your files without changing their filenames, or when you think Cloudflare is serving outdated files for whatever reason. If you have enabled Cache Everything, the Cloudflare cache will be purged automatically at the same time as the WP Rocket cache. Unless you have Cache …

Cloudflare indicates my website is slow

Okay, that’s a new information. I remember there were topics about WP Rocket + Cloudflare, therefore I suggest you to look up at the below articles in that terms: With the APO right? Kindly, consider looking into below article if you are going to combine WP Rocket and APO: If interested, a great bunch of related articles here: See at the below article for any helpful information WP Rocket + APO + Cloudflare: I would suggest below: memory_limit = 256M max_execution_time = 300 max…

[ccw]

@CroatiaHR

Thank you for your suggestions. However, these posts do not provide a solution for 404 status code.

[ccw]

With the help of Cloudflare support, I am finally able to solve the issue.

In CDN-Cache-Control · Cloudflare Cache (CDN) docs, it said " The Edge Cache TTL page rule overrides the amount of time an asset is cached on the edge (Cloudflare data centers). This page rule overrides directives in Cloudflare-CDN-Cache-Control/CDN-Cache-Control which manage how long an asset is cached on the edge."

So, to set different edge TTL for normal objects and 404, we should do as follows:

1. Remove all Edge Cache TTL settings.

2. In .htaccess, add the following line:

# Set edge TTL to 14 days for normal objects
Header set CDN-Cache-Control "max-age=1209600"
Header set Cloudflare-CDN-Cache-Control "max-age=1209600"

# Set edge to not cache 404
Header set CDN-Cache-Control "no-cache, must-revalidate, max-age=0" "expr=%{REQUEST_STATUS} == 404"
Header set Cloudflare-CDN-Cache-Control "no-cache, must-revalidate, max-age=0" "expr=%{REQUEST_STATUS} == 404"

After that, the edge will cache normal objects for 14 days and not cache 404, working as expected.

Thanks to all of your helps.

[simon]

Hi @ccw - glad you got this worked out and thanks for sharing the solution here - really helpful.