Hello,
It looks like there is a bug in how the cloudflare edge caching works regarding the HTTP Vary header. When a server serves a cacheable resource with a Vary: User-Agent header, cloudflare caches the resource when the first request comes, and then serves the same content again to all subsequent requests, independently of the User-Agent header of the request. This is very clearly violating the HTTP specification which states that :
An origin server might send Vary with a list of fields for two purposes: 1. To inform cache recipients that they MUST NOT use this response to satisfy a later request unless the later request has the same values for the listed fields as the original request (Section 4.1 of [RFC7234]). In other words, Vary expands the cache key required to match a new request to the stored cache entry.
I don’t think this can be called a feature if it clearly violates the HTTP specification…
The fact that this is a known problem and not considered a priority makes me a little worried about the service. Is there a list of known issues where cloudflare does not respect the HTTP specification ?
Not supporting the HTTP Vary header would be totally okay, cloudflare could simply forward the requests it does not know how to cache to the origin server. The problem here is that the requests are added to the cache, which means that the HTTP specification is not implemented correctly.
Probably a question for submitting to cloudflare ticket support. Cloudflare is on the forefront on bleeding edge technology so yes there are features that aren’t exactly to specification.
One that comes to mind is stale while revalidate header. Cloudflare only serves stale response on concurrent visitor request to the same stale resource and not when their are individual non-concurrent requests to same stale request. But technically the specifications are that states that the may serve the stale request and not must serve stale request on every request.
I upvoted your “product request” entry.
Cloudflare may be trying to do a lot of communication about being “bleeding edge”, but the HTTP Vary header is a standard basic HTTP header that was specified over 20 years ago. I find it a little disappointing that cloudflare didn’t implement it correctly.
Sounds like you’d like a custom cache key based on user agent. That’s available as an Enterprise feature. We don’t support the vary header per se, but you can write your own implementation in Workers on any plan type.
Sounds like you’d like a custom cache key based on user agent.
No, I would just like cloudflare’s proxy to respect the HTTP protocol. I am perfectly okay with not having access to some features such as custom cache keys, but I’m not okay with cloudflare breaking my valid web application by serving invalid contents to the wrong users. If the proxy does not support caching responses based on request headers, then it should simply not cache the request, instead of caching and serving invalid contents.
There are a number of CDNs which don’t support the Vary header because it is difficult to implement at best. You can disable caching for resources which require the vary header so that they are retrieved directly from origin utilizing page rules or workers.
How exactly would it be difficult to disable caching for responses that contain a Vary header ? This would already allow cloudflare to return valid HTTP responses to clients. Is there a policy about standard conformance ? Conforming to web standards would allow you to natively support more web applications without introducing subtle bugs that take time for your clients to solve. Is there a way to talk with an engineer working on this topic about this issue ?
I think you’ve made your point, enough now. Reiterating the same point over isn’t going to help. The community is trying to support you with alternative workarounds to your problem.
As suggested, raising a support ticket is the best way to talk to the Cloudflare team.
I’m sorry if I sounded annoying. I am thankful to other users who are trying to find workarounds. I was simply surprised and a little disappointed that cloudflare does not even consider that an issue, hence my repeated messages.
It would be difficult in the sense that a lot of origin software is going to set Vary, and if Cloudflare (and other CDNs) wouldn’t any cache Vary content, they’d very quickly end up not caching anything. Additionally, probably the most popular use of Vary is for Content-Encoding which they “should” ignore, fetching content once and recompressing it themselves.
Since Cloudflare is not a public forward proxy - they’re a reverse proxy acting on behalf of the server provider - bending the standard is OK. You understand the intricacies of Vary so this is a negative feature for you. But please consider that, regrettably, many others don’t, and this bending of the rules might be helpful to them.
To solve your specific issue, you can either set s-maxage=0 (in addition to Vary), to prevent Cloudflare from caching such content, or create a worker that instructs Cloudflare not to cache in the presence of Vary.