Possible to make large number of outgoing requests?

I would like to build a service that makes a large number (~50 000) of outgoing requests each day to fetch data from an external API. Currently this is technically possible if I have a worker endpoint that makes 50 outgoing requests (due to the subrequests limit) and invoke this endpoint 1000 times from outside of Cloudflare. It would be much nicer though if I could do all of this within Cloudflare, is this possible? I’m skeptical and can’t think of a solution due to the subrequests limit but maybe I missed something. I would really appreciate it to get any input on this.

For more context the use case is something like the following:

  1. Fetch a list from the external API that returns 1000 identifiers.
  2. For each identifier, make 50 additional requests from the external API to fetch additional data.

Hey, I’m on the Workers team. This is an interesting use of Workers as middleware, but unfortunately we currently can’t raise the subrequest limit (whenever a Worker leaves Cloudflare to fetch).

A potential workaround is you run a Cron Job in a Worker to grab those 1000 identifiers (over the course of a day) and store them in Workers KV. Then in a separate Worker with the same KV binding, you grab the identifier and make those additional fetch requests.

Would love to hear more about this use case if you want to send me a note: [email protected]

Hi @azhao, thank you for taking time to reply! I have been trying to work on a similar workaround today to try to get around the 50 subrequests limit per worker request. In the original post I changed the numbers a bit to try to make the problem simpler to illustrate. The actual numbers are around 11 000 identifiers, and 5-10 additional requests per identifier.

I thought I would try to make a chain of requests, where it guarantees that each request would not surpass the 50 subrequests limit. This is what I tried to do:

  1. Create an endpoint that fetches the identifiers and store them in Workers KV.
  2. Invoke a request with params ?start=0&end=${identifiers.length} in a new endpoint that either fetches additional data if end - start <= 5, or makes 5 new requests to itself with params ?start=0&end=1999, ?start=2000&end=3999 and so on.

I thought that if this would work then I would try to setup a Cron job to make the initial request like you mentioned. But I found out that Workers can not make request to their own endpoints. I did discover that it is possible for Workers to make requests to other Workers in different zones. So I think I could try to setup two Workers from different zones that bounce off each other until the final data fetching could be made. At this point the workaround is starting to feel too hacky and I’m not sure if it makes sense, or if it’s even allowed, to do all this just to get around the limitations for Workers.

I will definitely send you a message as well. Just wanted to leave these thoughts here if they could be helpful to anyone else. Thanks again!