Is it actually possible to upload to R2 buckets using wrangler?

Feedback Previews & Betas
#1

If you go to an R2 bucket in the dashboard it states the following:

Files larger than 300 MB can be uploaded using the [API](https://api.cloudflare.com/) or [Wrangler CLI](https://developers.cloudflare.com/r2/get-started/)

However as far as I can tell wrangler does not actually have file upload functionality

The only R2-related options in wrangler are to create/delete buckets or view the list of buckets

There seems to be no way to use wrangler to list the contents of a bucket or add files to or delete files from a bucket

Tried both wrangler1 (1.19.12) and wrangler2 (2.0.7), not much difference in this regard

If it’s actually possible to do this… how?

If it’s not possible, why does the dashboard say it’s possible?

overall experience with R2 so far is not great… took most of the day just to get to the point of being able to serve files HTTP using a worker (and that was by leveraging a third-party script from GitHub)

managing files via the dashboard is cumbersome and I haven’t yet found another viable way to do it

not really seeing a lot of value in using R2 versus just hosting the files directly on Cloudflare Pages, except for the ability to upload files greater than 25MB

#2

You can’t currently upload to R2 with wrangler - the team are aware of the wrong text on the dashboard.

You can use any other S3 compatible tool like rclone or the AWS SDKs as per the example docs. https://developers.cloudflare.com/r2/examples/

Using Pages as a “file host” isn’t the intended use and would be against the usual self-serve limitations of disproportionate non-HTML content if used for that.

What were the pain points that you ran into?

2 Likes
#3

It seems the aws cli also is unable to complete uploads.

For example, attempting this copy command:

$ aws s3 cp some_file.txt s3://my_bucket/ --endpoint-url https://my_account_id.r2.cloudflarestorage.com

Will result in this error:

upload failed: some_file.txt to s3://my_bucket/ An error occurred (NoSuchBucket) when calling the UploadPart operation: The specified bucket does not exist.

It’s unclear how to actually upload data to the bucket with e.g. aws cli.

#4

I’m trying to use rclone

[cloudflare]
type = s3
provider = Other
env_auth = false
access_key_id = (redacted)
secret_access_key = (redacted)
endpoint = https://(redacted).r2.cloudflarestorage.com
acl = private
bucket_acl = private

I can use “rclone tree” and “rclone ls” so I think my key is good (I can see files I uploaded via the dashboard):

$ rclone ls cloudflare:permanent/files/
  3070968 omedetou.mp4
 39841590 vaguely-organised-pictures.zip

however when I try to test by uploading an empty file it fails:

$ touch emptyfile
$ rclone -Pvv copy emptyfile cloudflare:permanent/files/
2022/06/08 12:01:41 DEBUG : rclone: Version "v1.50.2" starting with parameters ["rclone" "-Pvv" "copy" "emptyfile" "cloudflare:permanent/files/"]
2022/06/08 12:01:41 DEBUG : Using config file from "/home/username/.config/rclone/rclone.conf"
2022-06-08 12:01:42 DEBUG : emptyfile: Need to transfer - File not found at Destination
2022-06-08 12:01:42 ERROR : emptyfile: Failed to copy: s3 upload: 401 Unauthorized: <Error><Code>Unauthorized</Code><Message>Unauthorized</Message></Error>
2022-06-08 12:01:42 ERROR : Attempt 1/3 failed with 3 errors and: s3 upload: 401 Unauthorized: <Error><Code>Unauthorized</Code><Message>Unauthorized</Message></Error>
2022-06-08 12:01:42 DEBUG : emptyfile: Need to transfer - File not found at Destination
2022-06-08 12:01:42 ERROR : emptyfile: Failed to copy: s3 upload: 401 Unauthorized: <Error><Code>Unauthorized</Code><Message>Unauthorized</Message></Error>
2022-06-08 12:01:42 ERROR : Attempt 2/3 failed with 3 errors and: s3 upload: 401 Unauthorized: <Error><Code>Unauthorized</Code><Message>Unauthorized</Message></Error>
2022-06-08 12:01:43 DEBUG : emptyfile: Need to transfer - File not found at Destination
2022-06-08 12:01:43 ERROR : emptyfile: Failed to copy: s3 upload: 401 Unauthorized: <Error><Code>Unauthorized</Code><Message>Unauthorized</Message></Error>
2022-06-08 12:01:43 ERROR : Attempt 3/3 failed with 3 errors and: s3 upload: 401 Unauthorized: <Error><Code>Unauthorized</Code><Message>Unauthorized</Message></Error>
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 3 (retrying may help)
Checks:                 0 / 0, -
Transferred:            0 / 0, -
Elapsed time:       600ms
2022/06/08 12:01:43 Failed to copy with 3 errors: last error was: s3 upload: 401 Unauthorized: <Error><Code>Unauthorized</Code><Message>Unauthorized</Message></Error>

and yes my token is set to “Edit”

image
image1373Ă—79 9 KB

#5

and if I try to upload a larger file I get completely different errors:

$ rclone -Pvv copy largefile.7z cloudflare:permanent/files/
2022/06/08 12:14:12 DEBUG : rclone: Version "v1.50.2" starting with parameters ["rclone" "-Pvv" "copy" "largefile.7z" "cloudflare:permanent/files/"]
2022/06/08 12:14:12 DEBUG : Using config file from "/home/username/.config/rclone/rclone.conf"
2022-06-08 12:14:12 DEBUG : largefile.7z: Need to transfer - File not found at Destination
2022-06-08 12:20:32 ERROR : largefile.7z: Failed to copy: MultipartUpload: upload multipart failed
        upload id: AGpt+o5Z4mS4yj5vOwdqYWOyqmvzDT0hj4G1AiIMP7mFikD5PeplZ9L5gGVVJRqN0D+sn8oFTwebFwdGOcX8EqC4UEZ1YMfcBI22QT8a17UUlXQbxvr9WAU8bBNPV3a1T+IKIyIuLR5jEAXOG7VW0FH1nW7/PGbGqX8UPte/VMt+yyT3rFKEdCy85dl3ffzhwV71fSkhHHJaVJCEOiP8XJiNA+GG0+X2VZzgGCufRfu489z/j2APlmPLaw0Qsp2HEDTo5llwRl/lno0YseAdhZ/fRdi9KAuYADfiyxROfAdLV5DoxNHi5QpQcM6XPXsRxA==
caused by: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
        status code: 403, request id: , host id:
2022-06-08 12:20:32 ERROR : Attempt 1/3 failed with 3 errors and: MultipartUpload: upload multipart failed
        upload id: AGpt+o5Z4mS4yj5vOwdqYWOyqmvzDT0hj4G1AiIMP7mFikD5PeplZ9L5gGVVJRqN0D+sn8oFTwebFwdGOcX8EqC4UEZ1YMfcBI22QT8a17UUlXQbxvr9WAU8bBNPV3a1T+IKIyIuLR5jEAXOG7VW0FH1nW7/PGbGqX8UPte/VMt+yyT3rFKEdCy85dl3ffzhwV71fSkhHHJaVJCEOiP8XJiNA+GG0+X2VZzgGCufRfu489z/j2APlmPLaw0Qsp2HEDTo5llwRl/lno0YseAdhZ/fRdi9KAuYADfiyxROfAdLV5DoxNHi5QpQcM6XPXsRxA==
caused by: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
        status code: 403, request id: , host id:
2022-06-08 12:20:32 DEBUG : largefile.7z: Need to transfer - File not found at Destination
2022-06-08 12:25:30 ERROR : largefile.7z: Failed to copy: MultipartUpload: upload multipart failed
        upload id: AIq1FUhYXHm25rpg57tlRVmvf6bdHNijE0a3hUoAs37bmkAiul0nR74ysYMWO41mTf30k5MVGY51FoMSpAGLlbjfRdnmBeWFL3cCLsCFlMgcMy9RPLnFxubUk/cUdbFazO10BcGzJYgBaGv+ftNDsLG69Ms8c9+IsMwqw+BUD0iek0yh46rnO/tIxXar5Q6EgQ0/CYUVHY+NhbTaj6t4UxmyvbxoFBoWoRCMHn+epwzvt7JYNWkx/B9ptKW/XJ7MCZg7G8wjXPe/y3IbX4d0DsC213GIk9sY9Qcx1ItQz+PsLtk05ruiG6Z1IZ4KxfdnOw==
caused by: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
        status code: 403, request id: , host id:
2022-06-08 12:25:30 ERROR : Attempt 2/3 failed with 3 errors and: MultipartUpload: upload multipart failed
        upload id: AIq1FUhYXHm25rpg57tlRVmvf6bdHNijE0a3hUoAs37bmkAiul0nR74ysYMWO41mTf30k5MVGY51FoMSpAGLlbjfRdnmBeWFL3cCLsCFlMgcMy9RPLnFxubUk/cUdbFazO10BcGzJYgBaGv+ftNDsLG69Ms8c9+IsMwqw+BUD0iek0yh46rnO/tIxXar5Q6EgQ0/CYUVHY+NhbTaj6t4UxmyvbxoFBoWoRCMHn+epwzvt7JYNWkx/B9ptKW/XJ7MCZg7G8wjXPe/y3IbX4d0DsC213GIk9sY9Qcx1ItQz+PsLtk05ruiG6Z1IZ4KxfdnOw==
caused by: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
        status code: 403, request id: , host id:
2022-06-08 12:25:30 DEBUG : largefile.7z: Need to transfer - File not found at Destination
2022-06-08 12:28:48 ERROR : largefile.7z: Failed to copy: MultipartUpload: upload multipart failed
        upload id: AOh7a2I6SzgehZxsUyajcfYdY4fDeA3ld9Z2mCdggpjxRtHrTAeN/qYflPlI0pm0848JQ2IGxrlRtH/uFep4J+ryMBQYreT+X2eo64hhGDprPS03mbk2lPsByhn49zi0YiQbcciE4AK0cSuTW6zAM1XRcsj2K7IudmPn4cS7IEjTWvV9PdHTueqNbj3nLEjFeL8IkPYpuk4lh2wR6v3v7Z5U/ZJWK/6kKVcqC1eSwZEUbtCikyC+UecGsB1uE/avvEzMZsZpoH8gcU18lBUMsM/MXL2kncPPI0DRhDtQYw6OAKpRwQ2PU2xSNaekdWhRhw==
caused by: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
        status code: 403, request id: , host id:
2022-06-08 12:28:48 ERROR : Attempt 3/3 failed with 3 errors and: MultipartUpload: upload multipart failed
        upload id: AOh7a2I6SzgehZxsUyajcfYdY4fDeA3ld9Z2mCdggpjxRtHrTAeN/qYflPlI0pm0848JQ2IGxrlRtH/uFep4J+ryMBQYreT+X2eo64hhGDprPS03mbk2lPsByhn49zi0YiQbcciE4AK0cSuTW6zAM1XRcsj2K7IudmPn4cS7IEjTWvV9PdHTueqNbj3nLEjFeL8IkPYpuk4lh2wR6v3v7Z5U/ZJWK/6kKVcqC1eSwZEUbtCikyC+UecGsB1uE/avvEzMZsZpoH8gcU18lBUMsM/MXL2kncPPI0DRhDtQYw6OAKpRwQ2PU2xSNaekdWhRhw==
caused by: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
        status code: 403, request id: , host id:
Transferred:        1.050G / 1.050 GBytes, 100%, 1.228 MBytes/s, ETA 0s
Errors:                 3 (retrying may help)
Checks:                 0 / 0, -
Transferred:            0 / 0, -
Elapsed time:    14m35.6s
2022/06/08 12:28:48 Failed to copy with 3 errors: last error was: MultipartUpload: upload multipart failed
        upload id: AOh7a2I6SzgehZxsUyajcfYdY4fDeA3ld9Z2mCdggpjxRtHrTAeN/qYflPlI0pm0848JQ2IGxrlRtH/uFep4J+ryMBQYreT+X2eo64hhGDprPS03mbk2lPsByhn49zi0YiQbcciE4AK0cSuTW6zAM1XRcsj2K7IudmPn4cS7IEjTWvV9PdHTueqNbj3nLEjFeL8IkPYpuk4lh2wR6v3v7Z5U/ZJWK/6kKVcqC1eSwZEUbtCikyC+UecGsB1uE/avvEzMZsZpoH8gcU18lBUMsM/MXL2kncPPI0DRhDtQYw6OAKpRwQ2PU2xSNaekdWhRhw==
caused by: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
        status code: 403, request id: , host id:
#6

This is pretty outdated and official R2 support was added in the 1.59 beta.

#7

The current concurrent multipart restriction for R2 is 2 as opposed to S3’s default of 10.

Try setting your AWS config to include this under your R2 profile.

s3 =
  # default 10
  max_concurrent_requests = 2