I’m looking at supporting R2 as a storage backend for our platform, and for the most part, it’s been a drop-in replacement for S3. Happy with it, so far. One of the functionalities we have, is a media upload for end users, where we use Multipart uploads, sent directly from the end user to the bucket. I use Uppy as a basis in the browser side of things, which uses API calls to our platform to start/complete/abort the multipart and to presign the individual chunk URLs so that the end user can PUT to those.
This works, for the most part, but on larger uploads (I haven’t seen it with 2GB files but it’s pretty much a given on a 20GB file) there’ll be some 503 errors (no problem, those are automatically tried again), and eventually actual 500 errors from the R2 side. The 500 errors halt the upload, and there doesn’t seem to be much in the way of resuming it again. This may be a current limitation in Uppy, which I’ll look into, but in the mean time I’m somewhat surprised at the 500’s from R2. We’ve been trying for a couple of days now, so it’s not a short-term issue. It’s also not constant, I can upload a lot of files and they’ll all go swimmingly, it’s just the big one that fails.
I’d like to know what could possibly cause them, and a cursory search here tells me I’m not the only one that sees it. If there’s anything I can do to help troubleshoot, I’m all ears!
As an addition: we consistently get errors on a 20gb video. Smaller re-encodings all upload successfully, it’s as if there is an upper limit to the size of individual files. Is that the case?
That’s per chunk, not for the entire file. When uploading, Uppy cuts the 20GB file up in chunks that are way smaller than 5gb, so that’s not the limit we’re running into.
So whatever fits in 10k chunks, with a minimum of 5MB. So in my case, with a 20GB file, it’d use the minimum of 5MB, and send 2000 chunks, give or take.
It looks like Im running into this same issue. I ensure that only one file is uploaded concurrently and that the chunks are all the same size and no larger than 8MB.
I get random intermitted 500 errors “We encountered an internal error. Please try again.”. Retrying does usually work but this is not something I can quickly build into the software we use.