The time has come to offload all the images from our CMS to R2. There are about 360Gb of images or ~ 3 million of them.
What is the best strategy for uploading them from the website to R2. Has anyone been through this? In particular I’m keen to avoid any ‘sticker shock’ experiences from uploading too many images in too short a time frame and triggering some sort of S3 penalty. I know this is a real possibility on AWS, but I’m led to believe that R2 is better in this respect.
I’m OK with scriping an upload so that I upload newer images first and then go back in time to fill in the older images. But I don’t know how long 360Gb would take.
I see there is a facility to check R2 for an image, and if it doesn’t exist, then to pull it from another S3 bucket. But the original images will be coming from a webserver. Can I do the same trick with that.
Any advice, help, thoughts, received wisdom on the matter gratefully received.
Not loving this forum too much. CF support hasn’t answered after 15 days, and their suggestion to post a question in the forum yields only tumbleweed.
Well we went ahead and uploaded 460Gb of images to R2 using mainly rclone and a bunch of scripts, and didn’t hit any barriers in Class A or Class B operations. Waiting for first bill. Will let you know.
Appreciate the reply. I did try out Super Slurper. But as one of the buckets was in the EU region, it refused to sync the buckets. Go figure.
In the end I span up a temporary VM and used that to sync the files. Each file needs to download from source bucket to VM and then back up to target bucket.
I don’t have access to the billing records, so can’t comment on that aspect, but as my Manager hasn’t freaked out, I gather it wasn’t disastrously expensive.