- There’s only one KV - but you can use it via API or inside of Workers.
- Same limits apply.
- GraphQL and PostgREST isn’t the same thing, PostgREST is just a proxy to make calls directly to a postgres database on your origin server. GraphQL is a data fetching and caching layer that make requests to any existing REST back-end.
- Use Cloudflare Workers KV to store millions of tests that can be quickly run from Workers anywhere in the world.
Cloudflare Workers KV provides access to a secure low latency key-value store at all of the data centers in Cloudflare’s global network. Developers can use Cloudflare Workers and Workers KV to augment existing applications or to build entirely new applications on top of Cloudflare’s global cloud network. Workers KV scales seamlessly to support applications serving dozens or millions of users.
Here is an interesting project:
This is a very simple GraphQL API built on top of Cloudflare Workers KV. You may query and mutate items in a single Workers KV namespace by sending application/graphql format POST requests.
It seems possible to exceed the 1GB storage limit. It just costs extra - $0.50 GB/month. I wonder how competitive this pricing is.
- 1 GB - Key-value storage space
- 100,000 - Key-value reads per day
- 1,000 - Key-value writes per day
- 1,000 - Key-value deletes per day
- 1,000 - Key-value lists per day
- 1 GB - Key-value storage space
- 10M - Key-value reads per month
- 1M - Key-value writes per month
- 1M - Key-value deletes per month
- 1M - Key-value lists per month
- $0.50 GB/month - Additional storage
- $0.50/ 1M - Additional key-value reads
- $5/ 1M - Additional key-value writes
- $5/ 1M - Additional key-value deletes
- $5/ 1M - Additional key-value lists
The limits I’m talking about doesn’t have anything to do with storage, it’s about how much data 1 single worker request can process.
I am trying to find the 1MB on their website.
Accessible from all 250 Cloudflare locations
Supports values up to 25 MB
Supports keys up to 512 Bytes
Read and write from Cloudflare Workers
API to write to Workers KV from 3rd party applications
Uses Cloudflare’s robust caching infrastructure
Set arbitrary TTLs for values
Integrates with Workers Preview
Yes, you can process 25MB of JSON, but you have to use Workers Unbound (No CPU-limit). But that means you’ll have to pay for the traffic + CPU-time that you use.
Keep in mind, parsing 25MB of data doesn’t mean that it will consume 25MB of RAM, you’ll likely consume much more than the 128MB available in each request if you parse that amount of JSON or XML - just assigning that big of a string to a variable will likely cause you to run out of RAM.
What you CAN do, is store a 25MB file in KV and then stream that as a response directly to the browser (Without processing at all) - that will consume very little CPU-time.
Workers Unbound seems a promising alternative to AWS lambda. Unfortunately, I don’t see any tutorials. It would be good to get credits to be able to experiment and see how much a simple app costs.
Workers Unbound is like our classic Cloudflare Workers (now referred to as Workers Bundled), but for applications that need longer execution times. We are extending our CPU limits to allow customers to bring all of their workloads onto Workers, no matter how intensive.
07/27/2020 by Nancy Gao
Running a smallish Worker 40K times for 5 seconds each, cost under 1$. I’m sure you can afford it
Ok. So if I understand you correctly, I can use GraphQL with Workers KV as long as I am prepared to pay for Workers Unbound.
Your link to Cerulean’s unofficial Workers pricing calculator is very helpful. Thanks!
Built by Cerulean, unofficial Workers pricing calculator
Explaining the costs
Unbound is a bit more complex to calculate costs for and so I have built this tool to help people get a better understanding of the pricing model.
Requests: First 1 million requests are free, these are counted if they hit your Worker. With DDoS protection, any requests rejected by Cloudflare will NOT count towards this number. Even if your Worker has an error, it will count towards this number.
Memory: First 400,000-GBs is free. Each Worker you execute uses 128mb of memory, this is not changable yet. You are charged per millisecond the Worker runs.
Egress (Bandwidth): First 5GB free, costs $0.045 per Gigabyte sent to/from your Worker.
If you can live with a maximum of 6 concurrent connections (more requests will be queued) and only load data below RAM limit, then yes, you can. Keep in mind though that the GraphQL server for workers are at JUST below the max limit of 1MB per script. So it will consume a considerate amount of CPU and RAM just to boot up the request. I haven’t tested it myself, but I’d be surprised if it didn’t take 1-2 seconds before it would return responses.
The highest cost of all of this is usually KV writes, keep that in check and also the fact that you cannot update 1 KV value more than 1 time per second (not queued, simply denied).
A reference for the 6 concurrent connections limit
Simultaneous open connections
While handling a request, each Worker script is allowed to have up to six connections open simultaneously. The connections opened by the following API calls all count toward this limit:
fetch()method of the Fetch API
delete()methods of Workers KV namespace objects
delete()methods of Cache objects
Once a Worker has six connections open, it can still attempt to open additional connections. However, these attempts are put in a pending queue — the connections won’t actually be initiated until one of the currently open connections has closed. Since earlier connections can delay later ones, if a Worker tries to make many simultaneous subrequests, its later subrequests may appear to take longer to start.
If the system detects that a Worker is deadlocked on open connections — for instance, if the Worker has pending connection attempts but has no in-progress reads or writes on the connections that it already has open — then the least-recently-used open connection will be canceled to unblock the Worker. If the Worker later attempts to use a canceled connection, an exception will be thrown. These exceptions should rarely occur in practice, though, since it’s uncommon for a Worker to open a connection that it doesn’t have an immediate use for.
Yes, they are put in queue.
More on the RAM limit. The TransformStream API seems to offer a workaround.
Only one Workers instance runs on each of the many global Cloudflare edge servers. Each Workers instance can consume up to 128MB of memory. Use global variables to persist data between requests on individual nodes; note however, that nodes are occasionally evicted from memory.
If a Worker processes a request that pushes the Worker over the 128MB limit, the Cloudflare Workers runtime may cancel one or more requests. To view these errors, as well as CPU limit overages, go to Workers Open external link on the Cloudflare dashboard > Manage Workers > select the Worker you would like to investigate > scroll down to Invocation Statuses and examine Exceeded Resources .
Use the TransformStream API to stream responses if you are concerned about memory usage. This avoids loading an entire response into memory.
So far I’ve yet to see examples of someone using the TransformStream API on Workers, except for a few posts that kind-of-work but probably doesn’t.
But yes, the HTMLRewriter uses this, so it definitely works.
The problem isn’t to stream the data, that’s easy, it’s when to know how to parse chunks. Let’s say you want to parse JSON, how do you know that you have a chunk that contains valid JSON, so you can parse it? You can look for line-breaks, if it’s a JSONL file, but what if the response is a string without line-breaks? Same goes for XML or HTML.
TransformStream is definitely possible on workers and given the memory limitations, necessary in some cases. Ignore the comment about it not working on Workers, I run almost a hundred thousand requests a day through one of my async TransformStream workers.
This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.
Didn’t say it wasn’t possible, did say there’s no sample code to do it.
So, feel free to share.
I am new to Cloudflare so am curious to know how Tutorials get made. Who decides what topics will be chosen? Is there a way to request a topic such as using streaming?