Streaming a request


#1

Streaming http responses is pretty straight forward, however I am trying to struggling to stream an http request (body) to my origin server immediately. It appears that CF is buffering the entire request body, which makes a few things I’d like to do impossible.


#2

Hi @latency, would it be possible to post the worker script in question, or a simplified version of it? As long as your script is not reading the request’s body at all, we shouldn’t be buffering it.

Harris


#3

Hi @harris,

I can reproduce the request buffering with the script:

addEventListener('fetch', event => {
  event.respondWith(fetchAndApply(event.request)) 
})

async function fetchAndApply(request) {
  return fetch(request)
}

And then I upload a file through CF, and my origin will only see the request after the client has finished uploading it. If I circumvent CF, my origin will immediately be aware of the request while the client is in the process of uploading it.

To be clear, the problem isn’t in workers per-se – the full request buffering seems to be done by CF itself (quite possibly before it gets to the workers), I’m just hoping to avoid it (or rather: I need to. My server needs to be aware of uploads in progress, often done by slow mobile clients)


#4

Yeah… Whether you use Workers or not, Cloudflare itself buffers requests. This is intended to defend your origin against “slowloris” attacks. Of course, many modern web servers aren’t vulnerable to such attacks, so this may be a bit obsolete these days. Unfortunately, there’s no way to disable this at present.

An alternative option might be to use WebSockets. Workers currently support WebSocket pass-through, meaning that a worker which applies some rewrites to the URL or headers (but doesn’t modify the request or response body) should “just work” with WebSockets. (In the future we may also support terminating WebSocket in a Worker, although it’s tricky as there is currently no standard API for this.)


#5

Hmm, that’s a real shame. Would be great to be able to turn off request-body-buffering at the CF level :frowning: