Multiple Cache Results When Using cURL

Hey Everyone!

I’m seeing some very strange caching behavior from Cloudflare when using cURL to make requests. I have a worker setup that acts as a fetch proxy and cacheEverything is enabled. When hitting the route in the browser everything works as expected. A cache MISS then HIT with the same results until the cache expires.

However when hitting the same URL from cURL a cache MISS will be sent on multiple back to back requests. Then when cache HITs are returned a different cache is served on each request. When using cURL it seems like 3 or 4 versions of the cache is stored and served for some reason. The page in question is dynamically generated so that is a problem, there should only be one version of the cache.

I hope I was able to explain the issue clearly. Any help or a push in the right direction would be greatly appreciated.

Thanks!

That’s interesting, but not entirely surprising. If you’ve ever seen requests fluctuate between MISS and HIT, it’s because there are multiple servers at each edge node and they don’t sync their caches.

Is the content constantly changing?

You might consider Argo Tiered Caching, so requests within a region will more often pull from a central cache.

Thanks for the reply!

Regarding the edge nodes, this seems to only be happening when using cURL. If the url is hit from the browser everything works as expected. I’ll give Argo Tiered Cache a try but I just feel like there is some weird behavior happening when cURL is making the request. I’ve tried different User Agents etc but it’s always the same issue.

The backend server delivers JSON content that is dynamic on every request. I’m trying to cache a request for a period of time when the URL has certain parameters. But each time the backend is hit the response will be different.

That’s because there might be some additional info they can do a fingerprint for load balancing. Or even something to do with DDoS management as it’s not a normal browser’s user agent.

1 Like

Do you know of any settings I can toggle on the domain to test that out?

I can’t think of any way a user could override Cloudflare’s routing in a datacenter.

Excluding that, is it ok if different users in different parts of the world get different versions of the cache?

Did you see any improvement after enabled Argo Tiered Caching?

I gave Argo Tiered a try but it seemed to have no effect.