When copying a multiple files, R2 takes forever to upload the last few files

I’ve noticed when uploading multiple files (>= 50), the last file will take a very long time to upload. For example, here is a excerpt from rclone:

Transferred:      146.271M / 146.271 MBytes, 100%, 2.600 MBytes/s, ETA 0s
Transferred:          161 / 162, 99%
Elapsed time:        56.9s
Transferring:
 *                          my_last_file.jpg:100% /24.872k, 261/s, 0s

The previous 161 files uploaded without issue, but the last file is taking a long time to upload. Have other people run into this issue, what is the recommended solution?

1 Like

Greetings Vedantroy,

I’ve attempted to reproduce the error you are experiencing.

With 200 files of the size of 10MB each,
I didn’t experience the issue either on the Dashboard nor using rclone.

See screenshot below:

image

You can attempt the following without much technical expertise:

Uninstall and reinstall rclone, reconfigure it and try from the same computer.
Try using Rclone from an other virtual machine or desktop

This will help you identify if the issue is only affecting your computer/network.

I also have the same issue.

As you can see rclone copied 18557 files (~180 MiB/sec), then the rest stucked for maybe a half minute (that’s why average speed dropped down to 5.3 MiB/sec at the screenshot), then finishes the process. The files are correct, even the stucked ones: if I run again the rclone copy command, checksums are match, there are nothing to copy.

(Also there is the 502 error sometimes as you see, but rclone tries again at the end, then uploads correctly. It’s another issue, but I guess we do not want to discuss that in this tread.)

Code:
rclone copy do:bucketname/foldername r2:bucketname/foldername --exclude "**tmp**" -P --transfers=50 --fast-list --s3-upload-cutoff=100M --s3-chunk-size=100M --checksum

Any idea?