I am uploading large files (about 150Gi) to Cloudflare R2.
When downloading with CURL, the download defauls to HTTP/2. It downloads a few gigabytes, then fails with this error:
curl: (92) HTTP/2 stream 0 was not closed cleanly: INTERNAL_ERROR (err 2)
Forcing curl to use http1.1 with --http1.1
flag fixes the problem.
Note that I used the s3 CLI to upload the file, piping from stdin. I had to give an estimate of the file size, and this estimate is higher than the actual file size (I am unable to calculate an estimate beforehand). I am not sure if it is related.
Hi! Can anyone have a look at this? I still have an issue with downloading large (>100Gi) files from R2 such as “stream was not closed cleanly” or “connection reset by peer”.
I had previously said it only happens on http2, but http1.1 shows the same problem after all.
What are you making the HTTP requests to? Is it a custom domain?
Actually, it looks like it isn’t support based on a message from an employee on 2022-11-02
Yeah, looks like it’s not supported. That’s probably not something we’ll put much effort into in the near future, since I can’t find much information about clients actually allowing you to use http2. E.g. it’s difficult to find information about if boto even supports http2. I don’t think it’s likely that support for that is widespread, since AWS S3 doesn’t support http2 either as far as I can tell.
Message link
Thanks for your reply. I am simply using curl to download a file hosted on R2 on a custom domain. Curl uses http/2 by default.
However, I have the same problem when forcing HTTP 1.1 with curl --http1.1
So, simply put, the issue is that downloading a 100Gi+ file on R2 with curl often fails in the middle of the download.
You may try for yourself at https://polkashots.io
Hi Nicolas, I’d be interested in looking into this. Could you DM me in discord with more details about the file you’re trying to download? My Discord ID is Frederik#6268
As an aside, we’ve recently disabled http2, so clients should automatically use http1.1 now. But I understand that that doesn’t solve your problem.