Downloading KV data in bulk

I have an entire dataset stored in a single KV and need to migrate it to a more permanent storage facility (a proper database to enable querying). The service has been sitting online for about 2 years and has accumulated a considerable amount of keys, each under 100kb in size. I suspect there are may be upwards of 200k+.

Wrangler more the API have a method of bulk downloading the key and values. I’ve written a short worker to pull the data and output a JSON object:

addEventListener("fetch", event => {
  event.respondWith(handleRequest(event.request))
})

async function handleRequest(request) {
  const list = await playground.list({ limit: 50 });
  const output = {};
  for (let item of list.keys) {
    output[item.name] = await playground.get(item.name);
  }
  return new Response(JSON.stringify(output))
}

Unfortunately, the records seem to hang after a limit of 50 and to collect 200k records would require thousands of requests. Is there a more efficient/easier way to do this? If so I’m concerned about the cost of calling the worker and touching the KV n amount of times.

Suggestions are welcomed.

Thanks

There unfortunately isn’t really a better way that I know of, other than what you’re doing.

If you use an Unbound Worker, you should be able to hit about 1k records in a single request (999 to be specific, 1 for listing, and 999 get), as per the subrequest limits: Limits · Cloudflare Workers docs

And then you’ll have to hit that worker ~200 times to get your 200k records, with some kind of pagination.

1 Like

:frowning: this is one of the only times in all my Cloudflare experience that I’ve been let down by the Workers platform. This may not be a common use case, but having a way to bundle/move/export KV would be helpful.

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.