Asked around on discord with no replies, so moving here
What are the steps to reproduce the issue?
I can’t for the life of me figure out how to ask for general advice in this forum, so lemme know what’s the right place.
we have a couple hundred thousands of videos on cloudflare stream, and we want to back those up with another storage provider. for moving the videos, we are thinking on using cf workers and queues. i would be thrilled if someone can provide some reading material on the following broad questions:
how do i write a large scale oneshot operation with workers
what is the general pattern for dealing with pipeline errors in the cf ecosystem
how do i implement pausing and resuming of the pipeline
I’d suggest downloading each to some bucket such as recently Hetzner introduced cheap and reliable object-storage. Otherwise, Hetzner offers the Storage Box for “archive”, if that’s the case?
May I ask if you’ve tought about the procedures of what would the Worker need to do, one by one, scheduled, automatically, etc.?
Having a backup plan is indeed needed for your case, as well as any other customer using or depending on some online service(s) for videos such as Vimeo, YouTube, Stream, etc.
I’d suggest you to break down those in small steps.
Workers, I guess, could process your Stream videos in chunks.
Not sure how much CPU you’d have to use since Workers are limited by the memory, so we’d have to take care of not causing some issue and go step-by-step with the “job queue” since we’d have to think about the possible failure as well
With API, we could gather the list of videos being uploaded into the Stream.
My best guess, we could achieve this with KV, Queues and Workers.
You’d also have to consider monitoring such Worker.
