CF worker hangs from slow external API

I am using CF workers to import data from an external API. I leverage queues and durable objects to distribute load. I am assuming the probable case is that a single import request from a user will still hit the same isolate for worker and the durable objects it creates and thus end up sharing the same memory?

The external API is pretty slow, each request is anywhere from 600ms-2s (payload is big).

As an example, I have 22 pages, 40 items per page, each page is queued and processed by a new durable object. Each durable object makes a request to external API for page, retrieves 40 item ids, splits 40 items into batches of 10 item ids (max for next external API call), and creates a new durable object for each batch. Each batch takes from 500ms-1.5s.

I can’t tell if this suited for Cloudflare Workers, because I’m experiencing (locally) the request to end up hanging and eventually i gotta restart the server (around page 14). My laptop has plenty of memory, so my laptop can’t be the bottleneck?

I have been messing around with wrangler configurations and queue delays, but they are not making a difference. My assumption is that these DO subrequests are taking too long, but I can’t figure out how to know for sure what is happening.

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.