How to bulk read KV values via API or some other mechanism?

I have requirement fetching all of the data from a KV namespace. I’ve been using cloudflare python API getting list of all KV keys and getting values for each key. Usually this works fine if KV namespace has few entries. However my user generated KV namespace has hundreds of thousands of said key-values.

Obviously this resulting into extreme slow process of api calls to each KV key and hitting rate limit of 1200 calls per 5 min limit.

According to API, there is bulk write, bulk delete methods but not bulk reads. Is there a simpler process to get this data without the process of hours of api calls with rate limiting?

Is a custom worker dedicated to fetching data from KV possible without making this sensitive data public?

A worker with Access auth could work. Also some custom auth works, be it a password or ip limited.

2 Likes