I would like to build a service that makes a large number (~50 000) of outgoing requests each day to fetch data from an external API. Currently this is technically possible if I have a worker endpoint that makes 50 outgoing requests (due to the subrequests limit) and invoke this endpoint 1000 times from outside of Cloudflare. It would be much nicer though if I could do all of this within Cloudflare, is this possible? I’m skeptical and can’t think of a solution due to the subrequests limit but maybe I missed something. I would really appreciate it to get any input on this.
For more context the use case is something like the following:
- Fetch a list from the external API that returns 1000 identifiers.
- For each identifier, make 50 additional requests from the external API to fetch additional data.