R2 issue: uploading files (or chunks) to presigned urls

Hi all!
I am trying to upload a large file to R2 using multipart links. I have no trouble generating a presigned url where the chunk of the large file will be uploaded. But when I try to upload the chunk to the generated presigned url, I get the “Response status code does not indicate success: 401 (Unauthorized).” exception.
The same code works on other S3 platforms (e.g. backblaze).
Thanks!
Tadej

R2 has no presigned URLs support at the moment.

You could try something like this: R2 Presigned URLs · Denoflare

1 Like

Support for presigned URLs has just landed.

Is this something that could be used to make files accessible without using a Worker? Is there documentation?

Currently, refer to AWS S3’s documentation as it’ll be the same: Sharing objects using presigned URLs - Amazon Simple Storage Service

Presigned URLs have an expiry to them so they’re not quite perfect if you want always public access.

Oh cool it works with the latest rclone

$ rclone link cloudflare:permanent/files/omedetou.mp4
2022/06/17 16:14:26 NOTICE: S3 bucket permanent path files: Public Link: Reducing expiry to 1w as off is greater than the max time allowed
HTTP/1.1 200 OK
Date: Fri, 17 Jun 2022 21:15:21 GMT
Content-Type: video/mp4
Content-Length: 3070968
Connection: keep-alive
Accept-Ranges: bytes
Content-Range: bytes 0-3070967/3070968
ETag: "92e82543a46645d7211fad2dba06c42f"
Last-Modified: Sun, 05 Jun 2022 06:24:33 GMT
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Server: cloudflare
CF-RAY: 71cecccbaec3868a-ORD

I assume the “credentials” in the URL are just for the one file and thus are safe to give out?

The file name is included in the authentication signature, similar to the operation - i.e a presigned URL for GetObject will throw SignatureDoesNotMatch if you try do DeleteObject or change the key from, in your case, omedetou.mp4 to omedetou.mp3

Specifically: Authenticating Requests: Using Query Parameters (AWS Signature Version 4) - Amazon Simple Storage Service

Hi… I tried again but the upload still doesn’t work. There is one difference in the response. When I tried uploading before your post (that the presigned support has landed) I got an error 401 - Unauthorized, and now I get the error 400 (bad request).
Am I doing something wrong?
Thank you for the help!
Tadej

Possibly?

Without seeing the full response you get and what your request looks like, it’s hard to say.

Here’s the request…
2022-06-20 07:27:00.6451 FileWatcherIngester.Helpers.HttpClientHelper HttpClientHelper.Put call (url=https://4f6e9f957a39e1d840fc6fa4effcf365.r2.cloudflarestorage.com/ytv-upload-tmp/org_5/2022/06/252_b87fe0d3-7de6-4028-b227-a8064e9b1f5b.mp4?uploadId=AEA7tIsMQegW7Vhrfgdiz8nFFadBpQ257HVRr9zwXs8UA4ZpQLie/ovwRPnuZZkleV5SU6sVzmXSRyqY6bgurjbLkpqluq2dr78iYJWTDqS5mX2TgexOfDJ2f7LOurh/7R9nhPfSlrmtN6GERwYAhmhJx8dFj6AXJPocqCkg3AJHqVsbzugNv4o1KzRxSbIuXL5RKlEV9QumN9CPaMI8WbqQst133ie2AqpeEY7OF0EMwybRcqLX9ko6OCAQWpuPEembpe7Dv/DC4VUAbLNzii9paOlLHpHCe1FpSTm+3+8JohcCuL6sQa40BsLFCbTmQw==&partNumber=1&AWSAccessKeyId=5121d903f5de39702fffc07450e67d98&Expires=1655789217&Signature=nxolGp4kClBHgkBTS%2F4lSZFWrik%3D, try number=0, statusCode=BadRequest)
2022-06-20 07:27:00.6600 FileWatcherIngester.Helpers.HttpClientHelper EXCEPTION at HttpClientHelper.Put call (url=https://4f6e9f957a39e1d840fc6fa4effcf365.r2.cloudflarestorage.com/ytv-upload-tmp/org_5/2022/06/252_b87fe0d3-7de6-4028-b227-a8064e9b1f5b.mp4?uploadId=AEA7tIsMQegW7Vhrfgdiz8nFFadBpQ257HVRr9zwXs8UA4ZpQLie/ovwRPnuZZkleV5SU6sVzmXSRyqY6bgurjbLkpqluq2dr78iYJWTDqS5mX2TgexOfDJ2f7LOurh/7R9nhPfSlrmtN6GERwYAhmhJx8dFj6AXJPocqCkg3AJHqVsbzugNv4o1KzRxSbIuXL5RKlEV9QumN9CPaMI8WbqQst133ie2AqpeEY7OF0EMwybRcqLX9ko6OCAQWpuPEembpe7Dv/DC4VUAbLNzii9paOlLHpHCe1FpSTm+3+8JohcCuL6sQa40BsLFCbTmQw==&partNumber=1&AWSAccessKeyId=5121d903f5de39702fffc07450e67d98&Expires=1655789217&Signature=nxolGp4kClBHgkBTS%2F4lSZFWrik%3D, exception=System.Net.Http.HttpRequestException: Response status code does not indicate success: 400 (Bad Request).
at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
at FileWatcherIngester.Helpers.HttpClientHelper`1.Put(HttpClient client, String url, ByteArrayContent data, CancellationToken ct) in C:\code\repo\yawebtv\filewatcher\FileWatcherIngester\Helpers\HttpClientHelper.cs:line 118)

This is the wrong signature version - make sure that you’re using SigV4.

Hi…
I double checked… the signature version is 4.
(I am using the AmazonS3Client for c#, which has the default signature version 4)
regards,
tadej

That URL was most definitely SigV2.

SigV2:

GET /photos/puppy.jpg
?AWSAccessKeyId=AKIAIOSFODNN7EXAMPLE&Expires=1141889120&Signature=vjbyPxybdZaNmGa%2ByT272YEAiv4%3D HTTP/1.1
Host: awsexamplebucket1.s3.us-west-1.amazonaws.com
Date: Mon, 26 Mar 2007 19:37:58 +0000

Compare it to yours.

SigV4:

https://s3.amazonaws.com/examplebucket/test.txt
?X-Amz-Algorithm=AWS4-HMAC-SHA256
&X-Amz-Credential=<your-access-key-id>/20130721/us-east-1/s3/aws4_request
&X-Amz-Date=20130721T201207Z
&X-Amz-Expires=86400
&X-Amz-SignedHeaders=host
&X-Amz-Signature=<signature-value>  
1 Like

Now I managed to inforce v4, but now I get status code forbidden. I double checked and I am using the correct bucket name, key and secret. I would also like to reiterate that I am trying to upload a large file using multipartupload (uploading a chunk of file to the generated presigned url)…
here’s the request

2022-06-20 12:17:45.2018 FileWatcherIngester.Helpers.HttpClientHelper HttpClientHelper.Put call (url=https://4f6e9f957a39e1d840fc6fa4effcf365.r2.cloudflarestorage.com/ytv-upload-tmp/org_5/2022/06/41_9393c0f3-e6bc-40c7-a986-f019c4f6aceb.mp4?uploadId=AOiPWNB9zmg9nmgr+9mPaFJNdfL4IS1NsxvpXuAzQSkgSTpvs2mt5c7OrJhhX/5JsrpUiFIKG8lOj4Jkj95Q+7oECRRCTFmy6MrBbIbf3PBiiu0BOQbuwV3xHy02kqfurNz4Un/NsyZoLJXmVbOFhf8Cs+Xhz9VGVxox3/lIPgJLn4pwXJRP4VjRC2Qw5hB+vboUoKbNiKb7/sRnT0t9H24VADSV+eS1oXMHFeu0cbDmMN9P2+otMiPtjSLOpWWakO52HU5khw5+7Nlyu+ZQ4pSJnutJQubaqsUVFanZ4+1cFecNYeKoteZqp71c4ijxOQ==&partNumber=1&X-Amz-Expires=86400&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=5121d903f5de39702fffc07450e67d98/20220620/us-east-1/s3/aws4_request&X-Amz-Date=20220620T101740Z&X-Amz-SignedHeaders=host&X-Amz-Signature=12990c2d8114de58190fd2074adc4f5276437b329f6ecf7e9f7e358b83c64763, try number=0, statusCode=Forbidden)

I sent a reply but got the message that akismet blocked my post :slight_smile:
now I managed to enforce v4 and now I am getting statuc code forbidden. I am using the correct bucket name, key and secret (I double checked). I would also like to reiterate that I am trying to upload a chunk of a large file (multipart upload) to the generated presigned url…
Regards,
tadej

Without seeing the request or code, I can’t tell you why you’d be getting that error.

You’ll need to sign a CreateMultipartUpload operation, an UploadPart operation for each of your parts and then a CompleteMultipartUpload operation. Uploading and copying objects using multipart upload - Amazon Simple Storage Service

For request signing, multipart upload is just a series of regular requests. You initiate a multipart upload, send one or more requests to upload parts, and then complete the multipart upload process. You sign each request individually. There is nothing special about signing multipart upload requests.

In the event that you’re using a single presigned URL, you can’t use one for all of them. The operation is different in each and in the case of UploadPart, the part number is apart of the signature.

1 Like

The flow of multipart upload is correct (it works using backblaze as a s3 provider)…
here is the request
2022-06-20 12:17:45.2018 FileWatcherIngester.Helpers.HttpClientHelper HttpClientHelper.Put call (url=https://4f6e9f957a39e1d840fc6fa4effcf365.r2.cloudflarestorage.com/ytv-upload-tmp/org_5/2022/06/41_9393c0f3-e6bc-40c7-a986-f019c4f6aceb.mp4?uploadId=AOiPWNB9zmg9nmgr+9mPaFJNdfL4IS1NsxvpXuAzQSkgSTpvs2mt5c7OrJhhX/5JsrpUiFIKG8lOj4Jkj95Q+7oECRRCTFmy6MrBbIbf3PBiiu0BOQbuwV3xHy02kqfurNz4Un/NsyZoLJXmVbOFhf8Cs+Xhz9VGVxox3/lIPgJLn4pwXJRP4VjRC2Qw5hB+vboUoKbNiKb7/sRnT0t9H24VADSV+eS1oXMHFeu0cbDmMN9P2+otMiPtjSLOpWWakO52HU5khw5+7Nlyu+ZQ4pSJnutJQubaqsUVFanZ4+1cFecNYeKoteZqp71c4ijxOQ==&partNumber=1&X-Amz-Expires=86400&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=5121d903f5de39702fffc07450e67d98/20220620/us-east-1/s3/aws4_request&X-Amz-Date=20220620T101740Z&X-Amz-SignedHeaders=host&X-Amz-Signature=12990c2d8114de58190fd2074adc4f5276437b329f6ecf7e9f7e358b83c64763, try number=0, statusCode=Forbidden)

I’ve just successfully done a CreateMultipartUpload, UploadPart and AbortMultipartUpload using presigned URLs so I can only assume that there’s something in your code that isn’t compatible with R2.

As an example, with aws-sdk-php:

$cmd = $s3_client->getCommand('CreateMultipartUpload', [
    'Bucket' => $bucket_name,
    'Key' => 'ferriswasm.png'
]);

$cmd = $s3_client->getCommand('UploadPart', [
    'Bucket' => $bucket_name,
    'UploadId' => '<snip>',
    'PartNumber' => 1,
    'Key' => 'ferriswasm.png'
]);

// then to print the presigned URL for either
$request = $s3_client->createPresignedRequest($cmd, '+1 hour');

print_r((string)$request->getUri())

But have you tried uploading to the generated link? I successfully create the upload links, but then when I try to put some data on the generated links, I get status code forbidden…

Yes.

< HTTP/1.1 200 OK
< Date: Mon, 20 Jun 2022 12:10:10 GMT
< Content-Length: 0
< Connection: keep-alive
< ETag: "AERTAf217mWzrSiLAXh8d3NRTBmu20UiXD72x+9uUJcMZnIl3bclSPobXfx8I29REOqveKiD0UEiBI+iyDjk8lV2+s5alJrtg5mQKxziz54K7yYrRehvkb3rZkfubrPy4lT8Jh8OuF9ZYxVQx0wzrMmWo/ohrjmm0qwksipugvPV6N4xwZcjkanSPL7U9Q6nyYQyi6Xx952LGVge7SaQdcOa/9vqvpeFBGHBM0apRcLp8q8lUKFVMSrvp6+6TTvbJA=="

Without a reproducible example of your code, there isn’t any more suggestions I can offer.