I will be returning paginated responses using workers.
User may want to fetch ‘x’ hours/days of data, but data I store is sliced in one minute slices and stored as such in S3 cloud storage.
Inside worker I issue concurrently 10 requests to S3 (‘page’ size is 10 minutes) and then stream those responses similarly to https://developers.cloudflare.com/workers/recipes/streaming-responses/ example, but using native
pipeTo function of readable stream - that works great from my initial tests.
Client receives response and asks for the next ‘page’ (which is next ten minutes) and so on. What I’d like to do is
prefetch next page when serving current page so those will be in cache when requested again by the user. This way when streaming response is being returned, in the background ‘Cloudflare’ cache is populated for next page.
Is scenario like that supported or it’s something not encouraged and asking for the trouble?
What I basically would like to do is to issue 20 fetch (sub)requests for data, but stream only first 10, and ignore responses for next 10. Would that work?
Second thing I’d like to ask is about how to detect if streaming response failed due to for example resource limits etc. Files stored in S3 and returned are about 100kb -1000kb of compressed JSON data (up to 10mb when uncompressed, could be more) and I request and return ten of those in ‘single batch’ . I use
pipeTo which should not count for CPU limits, but still I’m not 100% sure if in some circumstances streaming such relatively large responses won’t cause resource quota errors and I’d like to detect that. Would such errors be even reported in workers UI given response had status ‘200’ but failed in the middle of straming?
Thanks a lot, I’d appreciate any tips.