When not in development mode image file is corrupt

My site is served by CloudFlare and HSTS is enabled. It has been working fine, but recently the logo file which is fetched by the following URL:

https://wiki.ekvastra.in/lib/exe/fetch.php/wiki/logo.png

is not loading. The browser says The image “…” cannot be displayed because it contains errors. If I enable development mode it works fine. I tried purging this file specifically, then purging everything, but it did not help.

Right now it does load fine. Is development mode off?

Interesting. Yes, development mode is off. The image does not load for me (13:51 IST). I have checked with Firefox, and Chrome after disabling browser cache on Windows 10 from India. And from a different ISP on my iPhone using Safari browser.

BUT when I checked from New Jersey using chrome on Windows Server 2012 it worked. So, the problem seems specific to my geo location? The webservers are located in Mumbai, India. I am in Hyderabad, India. From India Cloudflare returns 104.24.114.24 as the IP. From New Jersey Cloudflare returns 104.24.115.24 as the IP. Perhaps the edge server of Cloudflare specific to my geo location is having some issue?

It still is bypassing the cache for me though, which actually shouldnt be the case with that file extension.

Do you have any active page rules?

Yes, that shouldn’t be the case. I see cf-cache-status: HIT in the Header from Hyderabad, India. (But the image doesn’t load). From New Jersey I see cf-cache-status: BYPASS (and the image loads). From my control panel and audit log I can confirm that dev mode is disabled. Most recent audit log has this

{
“Enabled”: false,
“Name”: “dev_mode”,
“Type”: “caching”,
“Zone name”: “ekvastra.in”
}`

at Date: 2019-11-26T13:36:48+05:30

Yes, I have three page rules. But they are there from over three months. This behavior is from yesterday. The page rules are:

  1. www.ekvastra.in/
    Forwarding URL (Status Code: 301 - Permanent Redirect, Url: https://ekvastra.in)

  2. wiki.ekvastra.in/
    Forwarding URL (Status Code: 302 - Temporary Redirect, Url: https://wiki.ekvastra.in/doku.php)

  3. podcast.ekvastra.in/media/*
    Cache Level: Cache Everything, Edge Cache TTL: Custom 2419200 seconds

Nothing in these page rules should affect the caching of that particular URL.

https://support.cloudflare.com/hc/en-us/articles/200172516-Understanding-Cloudflare-s-CDN#h_bd959d6a-39c0-4786-9bcd-6e6504dcdb97 would suggest your origin is sending caching headers which prevent Cloudflare from caching it. Can you check that?

But that would still not explain what you seem to experience, that the cache appears to corrupt the file. Do you have Polish enabled for that domain?

I also suspected polish, but I never had it enabled. It has remained in disabled node. I have purchased few domain with Cloudflare but I am not on a paid subscription. Polish can be enabled only with Pro account.

This file used to be cached and served fine by Cloudflare until a few days back. Nothing changed on webserver, I did not even login to cloudflare until this problem appeared. I do purge my entire cache once in a fortnight or so for testing.

I enabled development mode now and the cf-cache-status is DYNAMIC and the image is coming up fine now. Here are the raw HTTP header that I have captured. You are right, apparently my webserver is setting pragma: no-cache. Though it doesn’t explain cf-cache-status HIT when I have dev mode disabled. x-xss-protection has wrong syntax, I will fix that, but it had been like this since ever and I do not think it will resolve my issue. I have left my setting in dev mode for next 3h after which it will auto-disabled by Cloudflare.

HTTP/2.0 200 OK
date: Tue, 26 Nov 2019 09:47:40 GMT
content-type: image/png
content-length: 7843
vary: Cookie
pragma: no-cache
expires: Thu, 26 Dec 2019 09:47:52 GMT
cache-control: public, proxy-revalidate, no-transform, max-age=2592000
content-disposition: inline; filename=“logo.png”;
accept-ranges: bytes
set-cookie: DW68700bfd16c2027de7de74a5a8202a6f=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0; path=/; secure; HttpOnly
x-frame-options: DENY
x-content-type-options: nosniff
referrer-policy: strict-origin-when-cross-origin
x-xss-protection: 1; mode=block; report=https://ekvastra.report-uri.com/r/d/xss/enforce
access-control-allow-origin: https://wiki.ekvastra.in
x-permitted-cross-domain-policies: none
report-to: {“group”:“default”,“max_age”:31536000,“endpoints”:[{“url”:“https://ekvastra.report-uri.com/a/d/g"}],"include_subdomains”:true}
nel: {“report_to”:“default”,“max_age”:31536000,“include_subdomains”:true}
feature-policy: camera ‘none’; geolocation ‘none’; microphone ‘none’; payment ‘none’
content-language: en, hi, sa
tk: N
content-security-policy: default-src ‘none’; object-src ‘self’; frame-ancestors ‘none’; base-uri ‘self’; connect-src ‘self’; form-action ‘self’; report-uri https://ekvastra.report-uri.com/r/d/csp/enforce; font-src ‘self’ https://cdnjs.cloudflare.com data:; style-src ‘self’ ‘unsafe-inline’ https://cdnjs.cloudflare.com; script-src ‘self’ ‘unsafe-eval’ ‘unsafe-inline’ https://cdnjs.cloudflare.com; img-src ‘self’ data:; manifest-src ‘self’
last-modified: Sat, 02 Jun 2018 02:06:33 GMT
etag: “93f66bf450e52d573dde3be35ca81e7b”
cf-cache-status: DYNAMIC
strict-transport-security: max-age=31536000; includeSubDomains; preload
alt-svc: h3-23=":443"; ma=86400
expect-ct: max-age=604800, report-uri=“https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct
server: cloudflare
cf-ray: 53baf13a0b8b7090-SIN
X-Firefox-Spdy: h2

HTTP/2.0 200 OK
date: Tue, 26 Nov 2019 09:48:45 GMT
content-type: image/png
cache-control: max-age=2678400
cf-cache-status: HIT
age: 2705
strict-transport-security: max-age=31536000; includeSubDomains; preload
alt-svc: h3-23=":443"; ma=86400
expect-ct: max-age=604800, report-uri=“https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct
vary: Accept-Encoding
server: cloudflare
cf-ray: 53baf2cfdf52d9b8-SIN
X-Firefox-Spdy: h2

Does your server IP address end in 51?

If so, it appears you are sending “proxy-revalidate”, which most likely prevents Cloudflare from caching. Can you remove that, then we could check what the issue with the cached file could be.

Yes, my server IP ends in 51. That header is getting set by the Dokuwiki software, let me fix it and then get back to you tomorrow. I won’t be able to make this change immediately. Thank you for the prompt responses.

Hi Sandro, it boiled down to cookie with expiry date. That was affecting the caching of the image. The image does not require any authentication/cookie so I removed the cookie now it is getting cached.

It took me several iterations narrow down to this. I tried removing the proxy-revalidate and some other cache parameters but it was the cookie. I can safely say this is a change from the Cloudflare side because the behavior was different earlier. But I am glad that Cloudflare has fixed it because the earlier behavior was wrong. I hope this can be documented somewhere.

Still BYPASS here.

You are right, I do get HIT now and the image is also coming fine … but I can’t explain why is it still BYPASS in other geo-location … irrespective of cookie. I am calling it a day … will ponder over it tomorrow. Please give me any directions if you get ideas. Thank you.

Well, my guess still is the caching header.

I left it overnight and now it is HIT again. Made no changes on webserver or Cloudflare. The difference is that I see that Cloudflare is now not sending the cooking that my webserver was sending earlier (because it is not on a separate cookie less domain). While there are other images (like apple-touch-icon.png) which are still in BYPASS (and having a cookie). I think there is some delay mechanism on Cloudflare once I purge cache and/or move out of dev mode? Again all this is from Hyderabad, India. I haven’t checked again from New Jersey.

Still BYPASS for me and there are only two cookies involved. One with a 34 byte long name, which only gets deleted and one “DokuWiki” one.

I somehow doubt it would be cookie related and it would be nice if the file could be cached, but the file does seem to get transferred just fine.

I have simplified the caching header to max-age 30 days.

Now I get HIT from India and corrupt file. The behavior has changed without any settings change from my end. New Jersey is still getting BYPASS.

I see more cookies when debugging with Chrome. It seems Cloudflare uses its own cross-site cookie to keep track of cache in some way? And there is a recent change in Chrome to not honor/send cookies that are cross-origin but does not have SameSite=None and Secure not set. This may be disrupting Cloudflare expectation. Whatever it is, I think there is something at Cloudflare end that can be investigated.

We are going a bit in circles I am afraid.

You have unproxied the record, so it generally is a bit difficult to say anything at this point. However, by forcing the Cloudflare proxy I still get a BYPASS.

As long as you dont fix that it is impossible to say whether the cache corrupts the file or not, though I doubt it.

I agree, my iterations make it difficult. I can confirm that I get HIT from Hyderabad and BYPASS from New Jersey at same time. This is not in my hands, it is something on the Cloudflare end. I promise not to touch Cloudflare or webserver or web-application for next 24h. I am sharing the output of wget --server-response ran from Hyderabad, India and New Jersey at the same time. Hyderabad says HIT and the NJ says BYPASS. HIT gives me 0 byte file. BYPASS gives me correct file. The Cache-Control is very simple now, it only has public and max-age.

[email protected]:/home/ekvastra> wget --server-response https://wiki.ekvastra.in/lib/exe/fetch.php/wiki/logo.png
Resolving wiki.ekvastra.in (wiki.ekvastra.in)… 104.24.114.24, 104.24.115.24, 2606:4700:30::6818:7318, …
Connecting to wiki.ekvastra.in (wiki.ekvastra.in)|104.24.114.24|:443… connected.
HTTP request sent, awaiting response…
HTTP/1.1 200 OK
Date: Wed, 27 Nov 2019 09:43:17 GMT
Content-Type: image/png
Content-Length: 7843
Connection: keep-alive
Set-Cookie: __cfduid=dac70096561f1fabf320ecacca0cf71ab1574847796; expires=Fri, 27-Dec-19 09:43:16 GMT; path=/; domain=.ekvastra.in; HttpOnly; Secure
Vary: Cookie
Expires: Fri, 27 Dec 2019 09:43:29 GMT
Cache-Control: public, max-age=2592000
Content-Disposition: inline
X-Frame-Options: DENY
X-Content-Type-Options: nosniff
Referrer-Policy: strict-origin-when-cross-origin
X-XSS-Protection: 1; mode=block
Access-Control-Allow-Origin: https://wiki.ekvastra.in
X-Permitted-Cross-Domain-Policies: none
Report-To: {“group”:“default”,“max_age”:31536000,“endpoints”:[{“url”:“https://ekvastra.report-uri.com/a/d/g"}],"include_subdomains”:true}
NEL: {“report_to”:“default”,“max_age”:31536000,“include_subdomains”:true}
Feature-Policy: camera ‘none’; geolocation ‘none’; microphone ‘none’; payment ‘none’
Content-Language: en
Tk: N
Content-Security-Policy: default-src ‘none’; object-src ‘self’; frame-ancestors ‘none’; base-uri ‘self’; connect-src ‘self’; form-action ‘self’; report-uri https://ekvastra.report-uri.com/r/d/csp/enforce; font-src ‘self’ https://cdnjs.cloudflare.com data:; style-src ‘self’ ‘unsafe-inline’ https://cdnjs.cloudflare.com; script-src ‘self’ ‘unsafe-eval’ ‘unsafe-inline’ https://cdnjs.cloudflare.com; img-src ‘self’ data:; manifest-src ‘self’
Last-Modified: Sat, 02 Jun 2018 02:06:33 GMT
ETag: “93f66bf450e52d573dde3be35ca81e7b”
CF-Cache-Status: BYPASS
Set-Cookie: DokuWiki=e3aa6bf1af1df32ddb5fe7a5864f7c69; path=/; secure; HttpOnly
Set-Cookie: DW68700bfd16c2027de7de74a5a8202a6f=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0; path=/; secure; HttpOnly
Accept-Ranges: bytes
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Alt-Svc: h3-23=":443"; ma=86400
Expect-CT: max-age=604800, report-uri=“https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct
Server: cloudflare
CF-RAY: 53c328297bf2e85d-EWR
Length: 7843 (7.7K) [image/png]

[email protected]:~$ wget --server-response https://wiki.ekvastra.in/lib/exe/fetch.php/wiki/logo.png
Resolving wiki.ekvastra.in (wiki.ekvastra.in)… 104.24.114.24, 104.24.115.24, 2606:4700:30::6818:7318, …
Connecting to wiki.ekvastra.in (wiki.ekvastra.in)|104.24.114.24|:443… connected.
HTTP request sent, awaiting response…
HTTP/1.1 200 OK
Date: Wed, 27 Nov 2019 09:47:27 GMT
Content-Type: image/png
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: __cfduid=d9b638d4d3b248cac176c9a6141a8aa5c1574848047; expires=Fri, 27-Dec-19 09:47:27 GMT; path=/; domain=.ekvastra.in; HttpOnly; Secure
Cache-Control: max-age=2678400
CF-Cache-Status: HIT
Age: 2054
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Alt-Svc: h3-23=":443"; ma=86400
Expect-CT: max-age=604800, report-uri=“https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct
Vary: Accept-Encoding
Server: cloudflare
CF-RAY: 53c32e46cc90d990-SIN
Length: unspecified [image/png]

Yes, I can confirm the cache control header.

There is one thing on the origin now, though, which I either wasnt there the first time I checked or I didnt notice and thats the “Vary: Cookie” header.

As far as I know Cloudflare only accepts Accept-Encoding in that header, however, as the article in question does not exist any longer (Google Cache) there is a chance something might have changed here.

For starters, I’d try to remove the “Vary” header to check if that could make a difference. However at this point it might also be a good idea to involve Cloudflare’s support. Maybe they can shed some light onto the BYPASS issue.

Yes, I am aware that Cloudflare does not support vary: cookie (except Enterprise account). And my usage of the web application does not really require this (I do not have any secure content that varies by logged in and logged users identified through cookies). I think I tested this before with Cloudflare (they simply ignore vary: cookie and assume vary: accept-encoding in its place, which is good for me). I am positive the caching used to work fine. I will change it to vary: accept-encoding and test once again after a day. But nothing so far explains HIT at one location and BYPASS at another location. I checked with HTTP2 also to rule out if Transfer-Encoding: chunked makes a difference but it doesn’t. I fee something inside Cloudflare is making a difference, something implemented recently.

Do I have the privilege to request help from support for a free account (two domain purchases) or do I have to purchase a subscription to request help from support?