In a Cloudflare Worker, how would you capture a file upload from a POST request and just store it in KV / best practice for storing images in KV?

What’s the best practice here?

Understand there’s some limitations and I’m experimenting and getting no where. I appreciate anyone’s help. Some code / context below (note, none of these are right and more trying to illustrate the issues…):

Try 1

It seems the following doesn’t work and strips the File data:

// I strip the File data so all you get is name="file_title.png"
const formData = await request.formData()

Try 2

Okay, so lets try something harder:

// I strip the name and mime type...
const formData = await request.arrayBuffer()

// Try to rebuild...
var rebuiltDataisThisEvenPossible = base64CrazyFunctionOrPolyFillMagic(formData);

Try 3

So finally why not just copy name/mimetype ideas from it KV Asset Handler and base it on the route:

// GET cat.png
run_kv_asset_handler_code()

// POST cat.png
let fileName = request.url.pathname // (not real code, none of this works)
let catUpdate = await KV_DEMO.put(fileName, await request.arrayBuffer())
console.log(catUpdate)
return new Response('check console')

Summary Questions

  • What’s the best practice for storing images/files in Cloudflare KV?
  • Limitations to getting Files from POST exist. If a Polyfill direction 1 or 2 rebuilding the ArrayBuffer to Base64 can potentially work, does anyone have an example of this? Is CPU time limts going to be an issue?
  • Currently sending images as Base64 pre-coded is my plan. Bringing this to the worker though would be awesome…

Disclosure: I am noob… Any dead-simple code is appreciated. Apologize if this is all over town

Thanks!

1 Like

@judge can probably answer this best, since this is exactly what he’s been experimenting with :wink:

Thanks @thomas4 for the response.

@Judge I actually came across your file hosting repo to steal ideas… It is a super cool project.

But correct me if I’m wrong, you actually don’t store any data via the Runtime APIs (aka, all from CL, not from KV.put(’…blah…’))? Your custom FileCore lib class is only for reading data.

Then for uploading, everything goes in via REST API as Base64 (and chunking it when huge).

You seem to skip the base64 option which I think makes sense for your chunking:

The bulk upload API has one more trick up its sleeve: not all data is a string. For example, you may have an image as a value, which is just a bag of bytes. if you need to write some binary data, you’ll have to base64 the value’s contents so that it’s valid JSON. You’ll also need to set one more key:

[
  {
    "key": "profile-picture",
    "value": "aGVsbG8gd29ybGQ=",
    "base64": true
  }
]

Workers KV will decode the value from base64, and then store the resulting bytes.

Anyway, your help would be INSANELY appreciated! Looking for any info / examples. I think I have looked at every single worker project on GitHub at this point.

Yeah, the state of uploading from a worker isn’t great. Your third example generally works if the data is passed non-form encoded (eg curl -X POST --data-binary @image.png https://something.workers.dev/upload). I was able to get it working from a browser input form by using javascript to send the file data itself as the body:

See: https://static2.judge.sh/upload (will only be up for a day or so since it uses my account’s KV). After upload, you can go to /<filename> to see the file you uploaded.

It’s probably possible to go above 10mb with this upload since you can split an arrayBuffer but I don’t have much time to check that at the moment.

2 Likes

Wooo, brings a lot of peace to my mind knowing my sample code was in the right ballpark. Thanks for the detailed reply with your experience from this.

I never realized that form-encoded vs non-form-encoded POST-ing was a thing (TIL!).

This all makes perfect sense to me and I think this would potentially work for my use case / experiment. I will run some test tonight on this and am thankful for code sample + the working sample.

For everyone’s future reference:

Judge’s Working Code (key parts extracted)

// POST (non URL form encoded) to e.g.: /upload
let targetname = event.request.headers.get("x-filename") // could I guess be based on a wildcard route too: /uploads/cat.png

let buf = await event.request.arrayBuffer()

await STATIC_KV.put(targetname!, buf)
1 Like

Feel free to share a complete worker for this for the #recipe-exchange when you’ve gotten it working, I think a lot would benefit from this! Or even as a template.

1 Like

That’s a great idea @thomas4 and will do. Haven’t had a chance to break this into a clean gutted demo yet. Just glad @Judge has SHOWN US THE LIGHT of what’s possible, lol

1 Like