Creating many D1 databases from inside Worker code

Hi, I’ve been trying D1 recently and I really like the idea of having an edge database with low write latency in one location and low read latency in many other locations without having to configure any replicas.

However, I doubt that D1 can be that in its current form. If all my users across the globe share the same database (or a small number of databases), write latency and write contention will be high for most users. Either that, or writes are asynchronously replicated to the master database, which would raise consistency issues.

So what I would like to do is create a separate database for each user or geographically colocated group of users from within my Worker code. I don’t want users collaborating with each other in SF to compete with my users in London for write access to a single database.

I’m hoping that at some point in the future D1 will let me create and use thousands (or millions) of databases on the fly rather than binding one or very few of them to a variable in my Worker environment. The write replica of each database could then automatically move to wherever most of the writes are coming from. It would also make security a whole lot easier to handle for multi-tenancy apps.

Cheers - x

Could you describe your use case in more detail?
Because it sounds a lot like Durable Objects and/or KV could fit your bill.

One of my near term use cases has smallish teams of users or individuals (think lawyers, journalists, researchers, financial analysts, etc) uploading documents to R2, extracting specific items from the documents and putting them into a database. Users can extend the data model, run reasonably complex queries, share the data with each other and create/publish reports. Teams will be geographically clustered in many cases.

This use case actually follows a classical pattern: Smallish, geographically colocated teams collaborating on (not necessarily big) data, then potentially distributing some data more widely. Every departmental line of business app works that way. Every SME SaaS solution works that way - accounting software, ERP, CRM, Payroll, etc. This is how ~90% of all databases on earth are used. Even using Excel/Google Sheets for these sorts of things is not unheard of :slight_smile:

KV and Durable Objects don’t work for me because they have no query engine.

1 Like

Oh, I hear you!

But I don’t think getting a proper SQL interface on workers should be a large problem.

And by their own admittance, Cloudflare built D1 on top of DOs. Maybe they’ll share some of their secret sauce and open source it for any worker based usage.
And if later D1 grows, migrating from one sqlite solution to another should be fairly straight forward as well. You could even right now replicate data from DO to R2 or D1 for more distributed or long running use cases, like building quarterly reports on a fixed dataset.