Issues with Streaming File Uploads to Cloudflare R2

Hello Cloudflare Community,

I’m currently working on a Node.js project where I need to stream file uploads directly to a Cloudflare R2 bucket. However, it seems like only the last chunk of the uploaded file appears in the bucket, or in some cases, the streaming process doesn’t work at all.

I believe similar functionality is possible with AWS S3 using the AWS SDK, and as R2 is compatible with the S3 API, I thought it should work. Below is a snippet of the code I’m using:

javascript

const AWS = require('aws-sdk');
const express = require('express');
const { PassThrough } = require('stream');

AWS.config.update({
    accessKeyId: 'MY_CLOUDFLARE_ACCESS_KEY',
    secretAccessKey: 'MY_CLOUDFLARE_SECRET_KEY',
    endpoint: new AWS.Endpoint('https://<MY_R2_ENDPOINT>'),
    s3ForcePathStyle: true,
    signatureVersion: 'v4'
});

const r2 = new AWS.S3();
const app = express();

app.post('/upload', (req, res) => {
    const passThrough = new PassThrough();
    const uploadParams = {
        Bucket: 'MY_R2_BUCKET_NAME',
        Key: 'FileName.ext',
        Body: passThrough
    };

    r2.upload(uploadParams, (err, data) => {
        if (err) {
            console.error("Error uploading:", err);
            return res.status(500).send('Upload error');
        }
        console.log("Upload successful:", data);
        res.status(200).send('Upload successful');
    });

    req.pipe(passThrough);
});

const PORT = 3000;
app.listen(PORT, () => {
    console.log(`Server is running on http://localhost:${PORT}`);
});

Questions:

  1. Does Cloudflare R2 support streaming uploads directly from an HTTP request as with AWS S3, or are there specific limitations or configurations needed?
  2. Are there any recommended best practices or alternative approaches for handling file uploads to Cloudflare R2?

Any advice, insights, or suggestions would be greatly appreciated.

Thank you in advance for your help!

Well, first of all, sorry about my English, I’m still not very good.

I had the same problem, and I opened your post here looking for solutions, but I managed to solve it with help from another place, and now that I was closing the browser pages and saw the post again, I came to post a solution that worked for me.

First I installed the @aws-sdk/client-s3 and @aws-sdk/lib-storage packages and then I installed the multer package to work with multipart/form-data content types, and then:

import express from 'express';
import multer from 'multer';

import { PassThrough } from 'node:stream';
import { Upload } from '@aws-sdk/lib-storage';
import { S3Client } from '@aws-sdk/client-s3';

const app = express();

// S3 client config
const s3Client = new S3Client({
  region: 'auto',
  endpoint: `https://${'R2_ACCOUNT_ID'}.r2.cloudflarestorage.com`,
  credentials: {
    accessKeyId: 'R2_ACCESS_KEY_ID',
    secretAccessKey: 'R2_SECRET_ACCESS_KEY',
  },
});

// Upload config
const uploadStream = async (fileUploadName: string, mimetype: string) => {
  const streamPass = new PassThrough();

  //Upload receives the client and upload params
  const streamPromise = new Upload({
    client: s3Client,
    params: {
      Bucket: 'R2_BUCKET_NAME',
      Key: fileUploadName,
      Body: streamPass,
      ContentType: mimetype,
    },
  });

  return { streamPass, streamPromise };
};

// Here is the custom multer storage
// It must contain the _handleFile and _removeFile methods 
class R2Storage {
  async _handleFile(
    req: Request,
    file: Express.Multer.File,
    cb: (error?: any, info?: Partial<Express.Multer.File>) => void,
  ) {
    const fileUploadName = randomUUID().concat(
      `_${file.originalname.replace(/ /g, '_')}`,
    );

    const { streamPass, streamPromise } = await uploadStream(
      fileUploadName,
      file.mimetype,
    );

    file.stream.pipe(streamPass);

    await streamPromise.done();
    cb(null, { path: `/${fileUploadName}` });
  }
 
  // If not used, it can return null normally
  _removeFile(
    req: Request,
    file: Express.Multer.File,
    cb: (error?: any, info?: Partial<Express.Multer.File>) => void,
  ) {
    cb(null);
  }
}

const storage = new R2Storage();
const parse = multer({ storage });

app.post('/upload', parse.single('file'), (req, res) => {
  try {
    res.send(path: req.file?.path);
  } catch (err) {
    console.log(err);
    res.status(500).send('Internal Server Error');
  }
});

app.listen(3000, () => {});

This solution works well here with express.

Awesome, thanks for coming back and posting your solution, I will test it in the coming days :slight_smile: