Chrome Accept header causing cache bypass and broken downloads since today

Hello, we are getting a very strange issue where files can be downloaded by Firefox but not by Chrome/Edge. This appeared out of nowhere some time today.

One of the links in question, routed to a Backblaze bucket behind Cloudflare using Transform Rules with Cache Everything set in Page Rules:
https://static.brickadia.com/launcher/1.5/BrickadiaInstaller.exe

After much experimentation, we tracked it down to a difference in the Accept headers sent by the browsers.

Chrome sends:
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Firefox sends:
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8

You can test these using curl:

This one (Firefox accept header) always works, returning valid file from cache:
curl -i --header "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8" https://static.brickadia.com/launcher/1.5/BrickadiaInstaller.exe

This one (Chrome accept header) does NOT work correctly, returning cf-cache-status BYPASS (!!!) and a 404 error.
Sometimes, for inexplicable reasons, it works, but if you run it 4-5 more times, it will start breaking again.
curl -i --header "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9" https://static.brickadia.com/launcher/1.5/BrickadiaInstaller.exe

For example:

Another example - requesting the identical url with Chrome Accept header 3 times in a row, two of them mysteriously got the BYPASS and failed:

Just to make sure the file actually still exists on the origin, I purged it from cache and did this again. After several requests that resulted in BYPASS and 404, one managed to work and return a MISS with the correct file.

Requesting the file with the Firefox Accept header always works properly.

I have tried extending the Transform Rule to delete the Accept header but this had no effect on it returning BYPASS and 404 most of the time when using the Chrome Accept header.

There is definitely something wrong here, on Cloudflare side. This issue appeared out of nowhere, after our config has worked fine for months. I believe it may be a bug caused by today’s launch of the Vary for images feature, but I am not certain.

I have also opened a support ticket for the same matter (2255096) but have not received a response yet.

1 Like

@smarsh you were very helpful in my last topic (this is that setup with the Authorization header added by Transform Rules to route to Backblaze) perhaps you could help get us some attention on this? It’s magically taken down our downloads unless people use Firefox…

To make absolutely sure this problem is on CF side and not the origin, I have also requested the file directly from Backblaze using the two Accept headers and it is working fine, yet when requesting with the Chrome Accept header through Cloudflare I get BYPASS and 404.

Authorization header snipped in the image is inserted by Transform Rules for the case where it goes through Cloudflare. That config had been working for months ever since it was made possible.

@zeblote apologies, just seen your message. Theres an ongoing incident which I believe is relevant to the issue you are seeing:

1 Like

We are not using Set-Cookie headers here, as seen in the response dumps. The downloads are seemingly breaking purely from the client sending the Accept header value that is used by Chrome. If instead we send the Accept header used by Firefox, it works fine. This can be replicated easily using curl.

Are you sure this is not a separate issue?

We’re implementing a fix right now, once its deployed can you re-check? The team feel that its related. If its still an issue post-fix then let me know and i’ll have them review.

1 Like

Sure, sounds good. I’ll keep watch on the status page and let you know if it works.

Fix was just implemented and is now rolled out globally. Can you re-test?

1 Like

Unfortunately it still doesn’t seem to work. I am getting more successful requests than previously, but there are still a number of fails where I get BYPASS and 404. For example:

This is using the Chrome Accept header curl command from the first message. When using the Firefox Accept header curl command, every request returns HIT and 200. I don’t know why it sometimes works and sometimes not. The client should never be getting a BYPASS cache status in response to this request.

Please just grab those 2 commands and run them like 50 times each and count how many 200 and how many 404 you get.

Firefox Accept:
curl -i --header "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8" https://static.brickadia.com/launcher/1.5/BrickadiaInstaller.exe

Chrome Accept:
curl -i --header "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9" https://static.brickadia.com/launcher/1.5/BrickadiaInstaller.exe

My results:

Firefox Accept: 50/50 successful
Chrome Accept: 31/50 successful

Are there any logs you can retrieve from the ray ids to see why some requests are working and some are failing? Did some deployment go wrong and only update half the servers in FRA datacenter? But that wouldn’t explain why other users in other parts of the world are also seeing this issue. Or why it suddenly started being broken around 24 hours ago. Really at a loss here!

According to your blog a new feature was deployed yesterday to support Vary for images which does something with the Accept header. So I am suspecting that this feature has a bug and is causing this issue. But obviously there isn’t any debugging I can do from my side to test that further.

The link in question is affected by this setup:

Maybe the issue can only happen when all of these are used in combination. Is there any way for you to use that specific link for reproducing and debugging it just in case?

Worth noting that we have not enabled Vary for images. The origin is sending a Vary: Accept-Encoding header, which according to the documentation, should not break the cache.

Looks like we have a few colo’s that havent fully upgraded yet, and you appear to be hitting one of these unfortunately (I just checked using your curl output).

Testing here in LHR and it works:

❯ curl -i --header "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9" https://static.brickadia.com/launcher/1.5/BrickadiaInstaller.exe
HTTP/2 200
date: Tue, 14 Sep 2021 15:53:36 GMT
content-type: application/x-msdownload
content-length: 71149086
x-bz-file-name: launcher/1.5/BrickadiaInstaller.exe
x-bz-file-id: 4_za81e37e2bb4c08a572ac0613_f108892a3494503c2_d20210716_m015343_c000_v0001055_t0011
x-bz-content-sha1: e39eacd84b6c6e1b96d5bf64857ca465ea187aa4
x-bz-upload-timestamp: 1626400423000
cache-control: public, immutable, no-transform, max-age=31536000
x-bz-info-src_last_modified_millis: 1626399468012
last-modified: Tue, 14 Sep 2021 01:14:03 GMT
cf-cache-status: HIT
age: 13572
accept-ranges: bytes
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=vE0yBohOAPoTIFHKFRqeQnhLwKI7IGgKYjC0o0Bi%2BxUVXQHE6DDl84qijFx9mFkvt7GXhnSEkTAfR2EhzfDqMCzWdEoPxuq08mOh1bbQGu9r4EOaArSXfGIRP2fg1mNwBBEO8vB5"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
strict-transport-security: max-age=31536000; includeSubDomains; preload
x-content-type-options: nosniff
server: cloudflare
cf-ray: 68eaca027c7fdc2b-LHR

Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.
❯ curl -i --header "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8" https://static.brickadia.com/launcher/1.5/BrickadiaInstaller.exe
HTTP/2 200
date: Tue, 14 Sep 2021 15:53:39 GMT
content-type: application/x-msdownload
content-length: 71149086
x-bz-file-name: launcher/1.5/BrickadiaInstaller.exe
x-bz-file-id: 4_za81e37e2bb4c08a572ac0613_f108892a3494503c2_d20210716_m015343_c000_v0001055_t0011
x-bz-content-sha1: e39eacd84b6c6e1b96d5bf64857ca465ea187aa4
x-bz-upload-timestamp: 1626400423000
cache-control: public, immutable, no-transform, max-age=31536000
x-bz-info-src_last_modified_millis: 1626399468012
last-modified: Tue, 14 Sep 2021 01:14:03 GMT
cf-cache-status: HIT
age: 13575
accept-ranges: bytes
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=q8U5RD5MtzDxalyEydokGfzd9OaJay3vFUzl4Ey7BKazbxCO5d9MVc6vIbYiLBAWg7UaviSuEyn4hQefiO39hoFiYYtrCV%2BkMr6wcDurcTUY1F6QYIXlCeXxHekVJbDR1SirUZn9"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
strict-transport-security: max-age=31536000; includeSubDomains; preload
x-content-type-options: nosniff
server: cloudflare
cf-ray: 68eaca18de6b214f-LHR

Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.
1 Like

That makes sense! I just used a VPN to change my location to Belgium and from AMS, 50/50 complete successfully. So we just need to wait a bit more :slight_smile:

:slight_smile: Sorry for the confusion. Let me know if you are still seeing problems in 1-2 hours time. The team are asking SRE to manually kick some of the servers in these colo’s to get them to upgrade.

Hmm, it still seems to be failing sometimes in FRA, but the amount of fails is less than previously.

Still giving errors even now in FRA. Are you sure deployment worked properly?

Hi, we’re still reverting changes and some of our servers are still left to be updated. Thanks for keeping us updated from your end!

2 Likes

Looks like we’re now fully deployed across those stubborn machines. Can you confirm no more errors?

2 Likes

I can no longer reproduce the errors in FRA.