I’m updating a KV from a Node server using the REST API.
When I look in the KV Dashboard I can see the value is updated pretty much immediately, since I imagine the Dashboard uses the REST API too and the KV must be in the same PoP. The problem is that when accessing the value from a Worker it takes a couple of retries until the updated value is available in the PoP where the worker is running.
I was under the impression a cold KV would simply take longer to read but it seems this is not true for updated values.
Do updated values get replicated automatically to other PoPs where this KV has been previously been read?
Is there something like a purge for a KV from all PoPs like we do with URLs?
“While writes will often be visible globally immediately, it can take up to 60 seconds before reads in all edge locations are guaranteed to see the new value.”
I’m not terribly worried about the delay in itself but about being certain a KV will be replicated without needing more requests from the KV in each and every PoP. Does that make sense?
Edit:
I’m asking because I’ve read a couple of posts here that mention that KV is pull based.
When you write to a KV in a worker it’s cached locally on that datacenter - which mean, you can read it immediately, but only if you use the same RayID cookie (Basically, same browser/client).
After that, the KV is sent to a central KV store and then each datacenter retrieve an update that the KV exists (Usually just a few sec).
When you then try to retrieve the KV from a datacenter that don’t have it, it just fetches it from the central location (slow). If the location is read often, then it’s also cached on that datacenter (faster).
I’ve only found FaunaDB to be reliable when writing and reading immediately despite location.
There is another option not mentioned (I think) in your 3 examples.
The KV already exists in the central storage and in some or all PoPs. Then it is updated via the REST API which, as far as I can tell, updates it in the central storage.
In that scenario, when would a KV be updated to other PoPs? Automatically to all PoPs after some seconds have passed? Or, after a worker in say LAX has used the stale local LAX KV copy and the system decides it should update the KV in LAX?
It seems the REST API updates it in central storage directly, the problem with updating via the REST API is that it’s rate-limited and doesn’t scale in the same way. I would only use that if you need to write few keys.
It’s been mentioned a few that times that it’s pull based only, so the central store doesn’t push data even when you use the REST API. Someone may want to confirm that though.
Ok let’s say I had a Worker for updating that KV instead of doing it via REST.
That wouldn’t really solve the problem of workers in other PoPs reading stale KVs until the system decides to pull an updated KV value to the Pop. Or would it?
When I tested by removing RayID cookies, the value was not immediately readable even though it got the same datacenter location, this suggest to me that the value is only cached by the current Worker Instance while it is being written to the central KV and then pulled - even to the same datacenter.
Also waiting for an API to purge KV. Right now you can only list 1,000 keys and do a bulk delete. We’re requesting 10x chunks of 1,000 and doing bulk deletes, but it’s not enough if the cache gets too large. The only way right now to properly purge cache is to just replace the reference and delete the old KV