Storing Compressed Data for Cloudflare Workers Sites

[nbabcock19]

aside: this was cross posted to stackoverflow before I realized that maybe here is more appropriate place for this question

My cloudflare workers site houses binary data that a react app fetches. This binary data is stored as gzip compressed as it compresses extremely well (we’re talking about a 20-25x reduction and uncompressed it is too large to fit in the 10MB KV limit). The problem I’m running into is that either the worker returns the data without the appropriate header:

Content-Encoding: gzip

or if I have the worker add the header, cloudflare will doubly compress the response. So how do I store gzip compressed data within cloudflare KV such that I can return it with the proper content encoding and without cloudflare doubly compressing the response?

Reference

For a minimal reproduction: here are the two worker scripts I’m using.

import { getAssetFromKV, mapRequestToAsset } from '@cloudflare/kv-asset-handler'

addEventListener('fetch', event => {
  try {
    event.respondWith(handleEvent(event))
  } catch (e) {
    event.respondWith(new Response('Internal Error', { status: 500 }))
  }
})

async function handleEvent(event) {
  const cacheControl = { browserTTL: 60 * 60 * 6 };
  return await getAssetFromKV(event, { mapRequestToAsset, cacheControl });
}

The above worker script returns the binary data without content encoding header, so the browser does not automatically inflate the response.

So then I tried adding the header manually via

import { getAssetFromKV, mapRequestToAsset } from '@cloudflare/kv-asset-handler'

addEventListener('fetch', event => {
  try {
      event.respondWith(handleEvent(event))
  } catch (e) {
    event.respondWith(new Response('Internal Error', { status: 500 }))
  }
})

async function handleEvent(event) {
  const cacheControl = { browserTTL: 60 * 60 * 6 };
  const resp = await getAssetFromKV(event, { mapRequestToAsset, cacheControl });
  resp.headers.set("Content-Encoding", "gzip");
  return resp;
}

The response has the correct content-encoding but cloudflare compresses the response so now it is serving gzip within gzip.

Is there any way to get a middle ground where I can serve compressed data from within cloudflare KV with the correct header and without doubly compressing the response?

[bulk88]

I dont have an answer. You might need to decompress the KV data in the CFW with wasm or zlib in pure JS lib, and pass it raw to respondWith/Response object. CFWs have restricted headers, “Host” and “Cf-Worker”, I think Content-Encoding is another one that the CFW runtime overrides from CFW to UA.

It is documented CF only uses uncomp or gzip to origin, but front end is brotli>gzip>uncomp. That would mean gzip from origin is uncompressed then recompresses as brotli on every request regardless if its a Worker or plain old orange cloud WAF/Cache. I do wonder if CF has ANY code path if a gzip/raw UA (no brotli), gets a gziped asset from origin, would the gzip file pass through CF POP without being decompressed? Or WAF fundamentally needs to unconditionally see the request body every time?

If my origin server sends out precompressed zopfli on-disk files (Accept-Encoding style, not a png), does CF recompress them to default of zlib #8 ( see Results of experimenting with Brotli for dynamic web content ) increasing their wire size? IDK. Someone should try it.

[nbabcock19]

This was answered on stackoverflow. The answer is to set encodeBody: "manual" like the following:

let resp = await getAssetFromKV(event, { mapRequestToAsset, cacheControl });

// Make a new response with the same body but using manual encoding.
resp = new Response(resp.body, {
  status: resp.status,
  headers: resp.headers,
  encodeBody: "manual"
});

// Modify headers and return.
resp.headers.set("Content-Encoding", "gzip");
return resp;