Random Cloudflare POPs return empty cached content, how can I check for this?

I have a Worker which caches a page in Cloudflare if certain conditions are met, otherwise it bypasses the cache and fetches from the origin.

It works well, but sometimes a Cloudflare POP will simply return a blank cached page. Accessing the page via other POPs works as expected. A user has reported an affected URL now from the Seattle POP which has allowed me to experiment further with this bug.

I checked my origin server’s Nginx logs and found it served all requests to the URL with a successful 200 response and the expected $body_bytes_sent. This confirms the problem occurred in Cloudflare’s Seattle POP.

Has anyone else experienced something like this, or does anyone have recommendations for mitigation? It seems Cloudflare’s cache cannot be trusted and I will have to come up with some checks on my own for this condition if I want to keep using it.

I am already checking for a successful 200 HTTP response code before storing pages into the cache, but that clearly isn’t the issue. My only other idea is to actually check the response body returned from the cache is not empty before sending it back to the user, but I am concerned that might introduce a small delay and increase the Worker’s TTFB.

This problem is particularly insidious in that I have no way of knowing when a given POP somewhere around the world is returning a blank white page for a URL (unless a user reports it) and there are no error messages or error status codes logged anywhere. If it happens in a major POP I could lose a lot of traffic to my site.

Never experience that, but I only cache 200 and if there is content in the request body. Do you enforce any cache logic? Also, I only push to cache with event.waitUntil(), after the user gets the response.

Never experience that, but I only cache 200 and if there is content in the request body. Do you enforce any cache logic? Also, I only push to cache with event.waitUntil() , after the user gets the response.

Yes. My code looks something like this:

addEventListener('fetch', event => {
  event.passThroughOnException()
  event.respondWith(handleRequest(event))
})

async function handleRequest(event) {
  const request = event.request

  // Build cache key
  const cache = caches.default
  const url = new URL(request.url)
  const cacheKey = url.origin + url.pathname

  // Check the cache
  let response = await cache.match(cacheKey)

  if (!response) {
    // If no response from the cache, fetch from the origin
    response = await fetch(request)

    // Do not cache an unsuccessful (non-200) response
    if (response.status != 200) {
      return response
    }

    // My cache logic
    // ....

    if (cache_this) {
      response = new Response(response.body, response)
      response.headers.append("Cache-Control", "s-maxage=604800")
      event.waitUntil(cache.put(cacheKey, response.clone()))
    }
  
  return response
}

@adaptive, can you share more details about how you are checking for “content in the request body”?

I’d be curious to see your technique for this as I figure out the best way to do it myself.