Etag and Content-Type support in KV Storage

Is there a way to store metadata for an item in KV Storage when posting? Specifically it would be great if you could store the content-type of the object and a ETag, so that we can serve 304’s easily.

I guess we could prepend the tags in the beginning of the files or solve it some other creative way, but would prefer not to if possible.

Thanks!

1 Like

Afaik, the only way to store “metadata” is to add it to the KV key name and then use the prefix to retrieve them.

See:
https://developers.cloudflare.com/workers/reference/apis/configuration_variables/kv/#listing-keys

Be aware of the higher costs to listing:

So to use this with storage, you’d have to roll your own File upload mechanism using the API, located here:

https://api.cloudflare.com/#workers-kv-namespace-write-key-value-pair

1 Like

We should have a good solution for you here sometime in the near future!

6 Likes

That sounds promising! Do you have a rough timeline on when you think it will be available? Would be great if we could wait for the proper solution instead of implementing a hack around it.

1 Like

Thanks @thomas4! That’s a nicer solution than what I came up with :slight_smile:

1 Like

I can’t make promises, but it’s more like “a month or so” than “six months or so”.

Hi @sklabnik!

I heard rumours that this fix might have been in the last release but can’t find anything about it. Do you know if it’s already out?

Best,
Markus

I am working on the release announcement right now, but yes, it is. The docs will be up for you hopefully soon! I’ll make a post in this thread about it when it happens.

6 Likes

:eyes:

Using this key + prefix trick to store meta and didn’t realize that list actually was independently charged.

Either way, super excited for this @sklabnik will be watching thread closely

Edit: Sneak peek found! This is such an awesome and much needed improvement for the community. Insanely appreciate you all expanding this service and looking forward to the official release.

3 Likes

Nice catch! I do hope we’ll be able to search in this Metadata when listing keys though.

1 Like

Looks cool! Wanted to try it out but not sure how the metadata is passed to the kv-storage rest api. Guessing it’s a header?

Not yet, but that would be a cool extension.

These docs have not been updated yet, but take a look at Cloudflare API Documentation, and imagine a new optional parameter, “metadata”, that has the JSON type. :wink:

There will still be a blog post coming soon but nice sharp eyes! To be clear, this feature is more general than headers; you can put any JSON you want into it. We expect most usage will be for headers though, but it wasn’t harder to make it more general and so we decided to do so.

2 Likes

Super cool… curious to see what the size limits are (like is 10mb = value + meta)

Enough questions, will wait and see what you got with the post! Congrats and thanks again

Hi @sklabnik,

I’m using KV-storage directly with the rest api and if I understand it correctly you can post metadata by appending a stringified JSON as a querystring with the metadata as key? It seems to work or at least not return any errors when I try it, but I can’t figure out how to to get the metadata back out… Do we need to add some header or querystring to the request?

I know the blog post will be published eventually but just so keen to try it out :slight_smile:

Right now, you’ll see them in lists, but we couldn’t change GETs due to backwards compatibility issues. The in-worker JS API has a call that lets you get them, but we haven’t rolled out an api.cloudflare.com API yet; most folks read from inside a worker, and so we prioritized that.

1 Like

Thanks for the reply @sklabnik!

I actually missed that there was an in-worker api, so thanks for the info :slight_smile: Is there a performance benefit using this api or is it just a wrapper?

So if I understand correctly I need to list the keys to get the metadata which would be a separate call from fetching the value?

The way I thought it would work would be that the metadata would be added to a header of the response… This way I could return a 304 in the worker as soon as I get the response object which from my perspective would be a simple solution. It it’s only available I might as well keep the metadata in a separate entry and make two requests right?
I’m considering adding the metadata to the value and the reading the first chunk, which is the most efficient solution I can think of, but it does get a bit messy so I would rather not go ahead with it. Would be some much cleaner if we just could read and write headers :slight_smile:

In general with Workers, if you use the in-worker API, it will talk to the local colo. But if you make a request to api.cloudflare.com, it’ll end up making an external network call. So in-worker APIs are generally better if you’re in a worker.

You make a separate request, but you get both back. It comes as JSON in the body, rather than as a header.

1 Like

Had a look on the documentation and it looks very promising :slight_smile: Thanks!

I am looking to send “ETAGS” with my response for my CloudFlare worker - I get penalty from GTMetrix because I am missing this, and penalty is not nice :wink:

Anybody who could guide me in the right direction, would be greatly appreciated.

Is/will it be possible to change just the metadata, without changing the KV value itself? A Worker may not have the latest value but may need to fill metadata.