Worker task queues?

The more I do testing with workers, the more I feel the need of task queues.

Most other serverless providers have this, do you plan on adding this to CF workers too soon?

1 Like

I’m actually using workers to roll-my-own queue, so, yes, I, too, am very interested in this topic. :slight_smile:

I love this idea! :crossed_fingers:

1 Like

We do have plans around this! Feel free to use this thread to chime in with any use-cases or requirements you have. What problems would you hope it solves?

4 Likes

A couple of cases where I currently need them:

  • Run scripts at specific times (Like a Cron job)
  • Queue work to be done in a specific order (For transactions)
5 Likes

Two examples that it would be great to move to Workers:

  1. Update reference tables used by validation functions. In the financial industry there are lots of validation functions based on public reference tables. Things like bank account check-digit algorithm registries. Workers and the KV are a natural except that the reference data must be updated on a regular basis. Currently there is no way to trigger a update worker. A simple cron facility for Workers would do the job.

  2. Sending webhook callbacks. Typically callbacks should be stored and then attempted some max number of times on an exponential backoff schedules until the response is 200 or the max attempt count is exceeded. There is no way to do this with Workers. To do efficiently is harder. Ideally you could set time triggers on WorkerKV keys. i.e. run this this script when this time comes around.

So I definitely need a cron like interface.

I need to batch stuff up (very quickly) and periodically send that batch somewhere.

I can batch stuff up quickly in a global variable, so if I could just call that worker by datacenter maybe.

Or if KV were fast I could query that on a schedule.

I can send requests to origin server and cache the page ahead of the time to save time and cache efficiently. Similar to Prefetching…

@zack any updates? i’m also interested in a way to trigger a worker - cron job

1 Like

We still have plans for this, but I can’t share details at this time.

i’m also interested in a way to trigger a worker - cron job

This is also something we’re considering for the future; it’s a useful feature for sure.

3 Likes

+1 for the CRON job. Is there somewhere I should be formally logging a vote for that?

Maybe @thomas4 can change the topic category to Feedback / Product Requests.

I’m unable to do that unfortunately, maybe CF admin can change it.

1 Like

Topic is too old, @thomas4 for the OP to recaterorise. CC @cloonan

1 Like

Nope, I hear you loud and clear; I track this personally, there’s no place for you to go other than what you just did :slight_smile:

2 Likes

+1 for this. We would like to create a few KVs which get updated every 15 minutes.

Got my vote on this too. Definitely would have use for it right now.

Lack of scheduling function invocations is why I had to go with google cloud functions.

I’m using a combination of Lambdas and Workers, most serverless platforms can easily complement each other until Workers gain the necessary features.

I was rather shocked at the insane management complexity of setting up lambdas compared to workers though:

Cloudformation (to manage the stack)
Cloudwatch logs (to log & monitor events)
EventBridge (to ingest events)
Kinesis (To ingest event data)
SQS (Event queue to manage deploys)
API Gateway (To setup a domain to use the API)
IAM (To manage permissions)
S3 (To store the Lambda code)

Compare that to the ~10 click and a paste that’s needed for Workers…

2 Likes

Having a lightweight scheduler similar to cron would be ideal to set weekly maintenance notices at the cloudflare, so that visitors get a friendly message, and traffic is diverted at the edge layer.

Otherwise you have to use another scheduler to interact with CloudFlare through the API - which is possible, but you introduce another system that has the potential to break. It would be nice to have it all in once place.

1 Like