My worker fetches a file from a server and uploads it to another server.
const response_1 = await fetch(server_1_url);
const response_2 = await fetch(server_2_url, {method: "PUT", body: response_1.body};
This works file for smaller files (100mbs) but randomly throws the following error :
413 Request Entity Too Large for large files.
After some digging, the only possible reason I could think of it is that the download_speed > upload_speed and thus it started filling up the ram.
(I am still open to thoughts on this…
)
const response_1 = await fetch(server_1_url);
const {readable,writable} = new TransformStream();
response_1.body.pipeTo(writable);
const response_2 = await fetch(server_2_url, {method: "PUT", body: readable};
This one works fine for all files but is relatively slow.
Am I doing something wrong…? Is there any better approach…? More importantly, Can I avoid using Transform Streams (slow 
) without hitting 413 error …?