Suddenly out of nowhere all our KV storages are throwing the following error: “KV GET failed: 500 Internal Server Error” We have not changed any of our code and this has been working for a long time already. Cloudflare Logs will not run either so we have no ability to see anything. Is KV down or something?
Yeah, KV died. It’s up again. Retroactive status page is probably coming.
We had 100% error rate for 20 minutes with the error messages being KV GET failed: 500 Internal Server Error between 18.35-18.49 UTC.
I can’t see a matching error it in the status dashboard.
Is there any way we could do to handle this or is retries and fallbacks handled by the platform?
Here’s the retroactive status page:
This was due to a complete KV outage. Nothing you could do besides retrying later.
Thanks for sharing. I find it remarkable that the status page is only retroactive and for a good 30 minutes you are in panic mode as nothing is working. Don’t they have people or software monitoring this so the status is real time? Very strange. Cloudflare is a multi-billion company.
1 Like
I’m also now wondering how reliable KV really is. I’ve made it a key aspect of my entire application and then something like this happens and I’m questioning my decision.
They do, internally it was a known issue before then, but the issue was only for 10 minutes total. There is a process to get status posts, and I would imagine just getting all of the details together / getting people to approve it took longer than resolving the actual impact.
It’s been at least months since the last impact like this. Cloudflare internally uses KV a ton, for the dashboard, for assets, for Cloudflare Pages, etc. A KV Incident is not small occurrence. I don’t have any exact statistics on me, but the last one I found was Cloudflare Status - Workers KV issues, and I know I wasn’t affected in that one and it was rather limited in scope.
OK, so that’s great to know that the company uses KV internally also, and they “eat their own cooking.” So they will catch problems early, presumably. Thanks for the info!
1 Like
This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.