Cannot seem to send multipart/form-data

Good day,

My final goal is sending a webhook message to Discord with files and embeds using cloudflare workers. I tried constructing the request in postman where it worked but not using the postman generated code within Cloudflare. In this example code I am using httpbin.org/anything so anyone can try it without generating a webhook. The form will be empty.

addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request))
})

/**
 * Respond to the request
 * @param {Request} request
 */
async function handleRequest(request) {
  var myHeaders = new Headers();
  myHeaders.append("content-type", "multipart/form-data");
  myHeaders.append("Cookie", "__cfduid=redacted; __cfruid=metoo");

  var formdata = new FormData();
  formdata.append("asd.png", "aaaaaaa", "asd.png");
  formdata.append("payload_json", "{\"content\": \"test\"}");

  var requestOptions = {
    method: 'POST',
    headers: myHeaders,
    body: formdata,
    redirect: 'follow'
  };

  // hookResponse = await fetch("https://discordapp.com/api/webhooks/fillmeinifyouwantotestwithawebhook", requestOptions)
  hookResponse = await fetch("https://httpbin.org/anything", requestOptions)
  return new Response(JSON.stringify(await hookResponse.json()), {status: 200})
}

How should I send it?

Thanks

Hi @diniboy,

Try removing this line:

myHeaders.append("content-type", "multipart/form-data");

Here’s an example, with some logging statements to help illustrate what’s going on: https://cloudflareworkers.com/#45af9b251560551955be6aa8fbf17965:https://tutorial.cloudflareworkers.com

The multipart/form-data MIME type depends on a “boundary” string to demarcate the boundaries between different parts of the form in the request body. The idea is to make the boundary string random enough so that it is unlikely to show up in the actual part payloads. This simplifies serialization/deserialization: the serializer and deserializer can read/write entire part payloads without worrying about having to escape any special characters.

The serializer (that is, the Request constructor) communicates the chosen boundary string via the Content-Type header’s boundary parameter, e.g. Content-Type: multipart/form-data; boundary=7f23f5fec10c01cd41b5c5ad4e80d250. Thus, the serializer must be in control of the Content-Type header.

Unfortunately, the Request constructor is also required to honor any user-provided Content-Type header. Thus, constructing a Request or Response from a FormData object and using a custom Content-Type header will always produce a broken object. :frowning:

This is obviously a rough edge of the API, and we should really emit a warning in this case.

Harris

Woah, never seen so quick, helpful and detailed reply. Tho the API is a bit weird to understand in this case, indeed.

Now I have a small issue if I would like to submit a file. According to the docs here: https://developer.mozilla.org/en-US/docs/Web/API/FormData/append#append_Parameters

The filename reported to the server (a USVString ), when a Blob or File is passed as the second parameter.

Since I am getting the attachment as another fetch call (it’s a simple png), and then try to call await myImage.blob() the console simply tells me:

Uncaught (in response) Error: Failed to execute 'blob' on 'Body': the method is not implemented.

:thinking: Is it not implemented? I can see it documented, here: https://developers.cloudflare.com/workers/reference/apis/request/#methods .

Hi @diniboy,

I’m afraid the Blob and File classes and their related methods are not implemented. We’d like to implement them, but it’s a bit tricky because it would be a breaking change. Supporting Files in FormData would require us to be able to return Files from FormData.get(), which presently (and incorrectly) always returns a string. This means that existing worker scripts which receive file uploads would see a behavior change from FormData.get() returning a string to instead returning a File. In other words, the change could break sites, if done without some sort of migration path in place.

For the time being, this means it’s effectively not possible to work with files containing arbitrary binary data (e.g., a png) in multipart/form-data. I’m sorry about that.

Harris

Ah, that’s pretty sad to hear since this way I cannot seem to upload files in multipart/form-data form.

Found an interesting project, specifically this: https://github.com/ssttevee/js-blob-ponyfill

But not sure if that’ll work or not.

So far I could manage to generate a vanilla JS script from the above linked project to gain Blob API support. This is what I got:

function mergeArrays(...arrays) {
    const out = new Uint8Array(arrays.reduce((total, arr) => total + arr.length, 0));
    let offset = 0;
    for (const arr of arrays) {
        out.set(arr, offset);
        offset += arr.length;
    }
    return out;
}

var _a;
/** @internal */ const $$buffers = Symbol('buffers');
/** @internal */ const $$merged = Symbol('merged');
/** @internal */ const $$size = Symbol('size');
const $$options = Symbol('options');
let encoder;
class BlobImpl {
    constructor(blobParts = [], options = {}) {
        /** @internal */ this[_a] = [];
        const { endings } = this[$$options] = options;
        if (endings !== undefined && endings !== 'transparent') {
            throw new TypeError('only transparent endings are supported');
        }
        for (let blobPart of blobParts) {
            if (typeof blobPart === 'string') {
                if (!encoder) {
                    encoder = new TextEncoder();
                }
                this[$$buffers].push(encoder.encode(blobPart));
                continue;
            }
            if (blobPart instanceof Uint8Array) {
                this[$$buffers].push(blobPart);
                continue;
            }
            if (ArrayBuffer.isView(blobPart)) {
                blobPart = blobPart.buffer;
            }
            if (blobPart instanceof ArrayBuffer) {
                this[$$buffers].push(new Uint8Array(blobPart));
                continue;
            }
            if (blobPart instanceof BlobImpl) {
                this[$$buffers].push(...blobPart[$$buffers]);
            }
        }
        this[$$size] = this[$$buffers].reduce((size, buf) => size + buf.length, 0);
    }
    get size() {
        return this[$$size];
    }
    get type() {
        return this[$$options].type || '';
    }
    /** @internal */ get [(_a = $$buffers, $$merged)]() {
        return mergeArrays(...this[$$buffers]);
    }
    slice(start = 0, end = this[$$size], contentType = '') {
        if (start < 0) {
            start = Math.max(this[$$size] + end, 0);
        }
        else {
            start = Math.min(start, this[$$size]);
        }
        if (end < 0) {
            end = Math.max(this[$$size] + end, 0);
        }
        else {
            end = Math.min(end, this[$$size]);
        }
        if (Array.from(contentType, (_, i) => contentType.charCodeAt(i)).some((c) => c < 0x20 || 0x7e < c)) {
            contentType = '';
        }
        else {
            contentType = String.prototype.toLowerCase.call(contentType);
        }
        const newParts = [];
        let offset = 0;
        for (const buf of this[$$buffers]) {
            const sum = offset + buf.length;
            if (sum < start) {
                continue;
            }
            if (!newParts.length) {
                if (sum < end) {
                    newParts.push(buf.slice(start - offset, end - offset));
                    break;
                }
                newParts.push(buf.slice(start - offset));
            }
            if (sum < end) {
                newParts.push(buf.slice(0, end - offset));
                break;
            }
            newParts.push(buf);
        }
        return new BlobImpl(newParts, { type: contentType });
    }
    stream() {
        return new ReadableStream({
            type: 'bytes',
            start: (controller) => {
                for (const buf of this[$$buffers]) {
                    controller.enqueue(buf);
                }
                controller.close();
            },
        });
    }
    text() {
        return Promise.resolve(new TextDecoder().decode(this[$$merged]));
    }
    arrayBuffer() {
        return Promise.resolve(this[$$merged].buffer);
    }
}

addEventListener('fetch', event => {
    event.respondWith(handleRequest(event.request));
  });
  
  /**
   * Respond to the request
   * @param {Request} request
   */
  async function handleRequest(request) {
    var myHeaders = new Headers();
    myHeaders.append('Cookie', '__cfduid=redacted; __cfruid=metoo');
    image = await fetch('https://http.cat/100');
  
    var formdata = new FormData();
    formdata.append('asd.png', new BlobImpl(image, {type: 'image/png'}), 'asd.png');
    formdata.append('payload_json', '{"content": "test"}');
    
    var requestOptions = {
        method: 'POST',
        headers: myHeaders,
        body: formdata,
        redirect: 'follow'
    };
    
    aaa = await fetch('https://httpbin.org/anything', requestOptions);
    return new Response(JSON.stringify(await aaa.json()), {status: 200})
  }

I assume instead of new BlobImpl(image, {type: 'image/png'}) I should have new BlobImpl(await image.text(), {type: 'image/png'}). Well, it throws a warn that the representation might be wrong then, though it seems to be able to produce a valid representation of the image. Though in this case my worker gets a timeout. I guess because this impl is too slow for my image.

Any workarounds? I personally wouldn’t care if it would be extremely hacky, but it’s really disappointing me that I can’t solve that. Discord’s webhook API works the way as it is. And in order to be able to post images, I must use this format.

Due to worker CPU limit, processing files over 2M is usually not possible. Images can be hard to process even at 500kb size. Using a WASM might help a little bit.

In general though, image processing or large file processing is not possible.

I’m afraid I don’t have any easy workarounds. It’s not possible to use the existing FormData class since its interface only uses strings, so any binary data would have to be able to survive a UTF-8-decoding/encoding.

In theory, since it’s possible to get the raw bytes of the multipart/form-data body via request.arrayBuffer(), and it’s possible to provide raw bytes as the body in a Request constructor, it should be possible to avoid using the FormData class and instead parse the multipart/form-data manually, do whatever processing you need, then reassemble the multipart/form-data. Depending on what you need to do with the PNG, though, you could easily run into the CPU limit, as @thomas4 mentions.

I see. I basically don’t have to do any manipulation of the data, all I am doing is making a request to a picture and trying to send that over to Discord as a Discord embedded image in a message. Could you show me an example of fetching an direct image (could be any png or jpg on the internet) from an URL sent with the multipart form data?

Thanks

If you’re communicating with Discord using a worker, this might be good to know:

I saw that though that doesn’t affect me. I am trying to send a webhook message which is still not blocked by Discord. Only the CDN content retrieval.

2 Likes