Can't set "Content-Length" for GET requests

Developers Workers
1

Hi

I’m trying to store binary data (+100MB by splitting the data in 10MB chunks) in Workers KV, and it works very well, but I can’t override the Content-Length header. Is that expected? I want the Content-Length header so the user can track the download progress.

Minimal working example:

addEventListener('fetch', event => {
  event.respondWith(handleRequest(event))
})

async function handleRequest(event) {
  let { readable, writable } = new TransformStream()

  let response = new Response(readable) 
  response.headers.set("Content-Length", 5000) // foobar has a size of 5000 bytes
  response.headers.set("Content-Type", "application/octet-stream")
  stream(writable)

  return response;
}

async function stream(writable) {
  let readable = await DATA.get("foobar", "stream")
  await readable.pipeTo(writable)
}

If I issue a HEAD request Content-Length is correctly returned:

$ curl -I 'https://xxx.workers.dev/'
HTTP/2 200 
date: Sun, 17 May 2020 15:55:22 GMT
content-type: application/octet-stream
content-length: 5000
set-cookie: __cfduid=d6b7e242e522c799a5ec71003d730e8a91589730922; expires=Tue, 16-Jun-20 15:55:22 GMT; path=/; domain=.fffffffff.workers.dev; HttpOnly; SameSite=Lax
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
cf-ray: 594e85b6980f736b-CPH
cf-request-id: 02c4f1e61c0000736beeb22200000001

but if I issue a GET request it isn’t:

$ curl -I -X GET  'https://xxx.workers.dev/'
HTTP/2 200 
date: Sun, 17 May 2020 15:56:47 GMT
content-type: application/octet-stream
set-cookie: __cfduid=dd2135bf3e2416154474bc87eff200d8d1589731007; expires=Tue, 16-Jun-20 15:56:47 GMT; path=/; domain=.fffffffff.workers.dev; HttpOnly; SameSite=Lax
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
cf-ray: 594e87cacce2735f-CPH
cf-request-id: 02c4f332bf0000735f803fb200000001
2

Hi Kristian,

My understanding is that when you’re using chunked encoding you’re not allowed to pass a content-length header as you can’t be certain how long the content will be. My experience is in general that Cloudflare always will override any content-length headers you try to pass which probably is a good thing as it’s easy to shoot yourself in the foot.
What the browsers typically do when downloading content is to first make a head requests or a range request from 0-1 to get the length of the file and after that request the entire file.

1 Like
4

I’m not using chunked encoding, but the data is streamed, so the length can’t be predicated.

That sounds plausible, but in my case I need it :slight_smile:

Are you sure? I have never seen that behavior in Firefox or Chrome.

5

It does seem like I’m wrong :slight_smile: Pretty certain that the content that was streamed through cloudflare was using chunked encoding before, but when I test now I’m getting normal range requests back with a content-length. Nice!

6

@kristian Have you managed to solve this? I just encountered the exact same issue.

7

Sadly I haven’t. I did do some more research though and I think it is working intended and it is a consequence of the way streams is implemented/specified: Streams based uploading with Content-Length (not chunked encoding) · Issue #95 · whatwg/fetch · GitHub

8

You can use FixedLengthStream to set Content-Length.

https://developers.cloudflare.com/workers/runtime-apis/streams/transformstream/#fixedlengthstream