Decryption while streaming

I’ve been working on building a worker to decrypt files (they are “encrypted at rest” on S3 storage) using a CloudFlare Worker, so as to do the decryption as close to the client as reasonably possible. The files are encrypted when written by PHP app (on origin servers). The file written includes the IV, HMAC and base64 encoded payload.

For small files, this can be done by loading the entire file into RAM, then processing it entirely in RAM, then sending the decrypted content to the browser. But with RAM limits and the number of copies in memory, it’s not long before running into files that are too big to handle this way. So, I’m trying to decrypt while streaming.

Here’s the relevant part of the code (from my decryptStream () function):

    // Buffer size needs to be a multiple of the cypher block size.  For AES, the block size is always 128 bits.
    const base64CyphertextBufferSize = 128;
    const cyphertextBufferSize       = base64CyphertextBufferSize * 6 / 8;
    let   base64CyphertextBuffer     = new Uint8Array (base64CyphertextBufferSize);
    let   cyphertextBuffer           = new Uint8Array (cyphertextBufferSize);

    // Set up for decryption.
    const passwordHash    = await generate_key_from_password (password);
    let   algorithm       = {name: 'AES-CBC', length: cypherKeySize};
    const key             = await crypto.subtle.importKey ('raw', hex2uint8array (passwordHash), algorithm, false, ['decrypt']);
    algorithm             = {name: 'AES-CBC', length: cypherKeySize, iv: iv};
    let   lastCryptoBlock = hex2uint8array (iv);

    try
    {
        let loop_counter = 0; // For debugging.
        // Decrypt as long as there is content flowing.
        for (;;)
        {
            loop_counter++;
            const { done }   = await reader.read (base64CyphertextBuffer);
            cyphertextBuffer = string2uint8array (base64_decode (utf8decoder.decode (base64CyphertextBuffer)));
            algorithm.iv     = lastCryptoBlock;
            decryptedChunk   = await crypto.subtle.decrypt (algorithm, key, cyphertextBuffer);
            await writer.write (decryptedChunk);
            lastCryptoBlock  = cyphertextBuffer.slice (cyphertextBufferSize - (cypherBlockSize / 8))

            // Are we at the end of the input?
            if (done)
            {
                break;
            }
        }
    }
    catch (error)
    {
        console.error ("decryptStream () exception while \"try\"-ing the \"infinite\" loop; ignoring.\n", error);
        // Ignore the exception.
    }

    try
    {
        await writer.close ();
    }
    catch (error)
    {
        console.error ("decryptStream () exception while \"try\"-ing to close the writer; ignoring.\n", error);
        // Ignore the exception.
    }

Obviously, there is a bunch of other code to set this up, but that’s all good. This is the setup and the loop that will do all the decrypting. The basic structure to get to this is:

addEventListener ('fetch', event => { return event.respondWith (handleRequest (event.request)); });

async function handleRequest (theRequest)
{
    // Do a bunch of setup, authentication, etc.
    // Fetch something from the origin server, then decide we have a response we can process.
    return processOriginResponse (originResponse);
}

async function processOriginResponse (originResponse)
{
    // Prepare the responseInit object.
    let responseInit =
        {
            headers:
                {
                    'Content-Type': originResponse.headers.get ('Content-Type'),
                    'Content-Disposition': originResponse.headers.get ('Content-Disposition'),
                    'MyFiles-Encrypted': true,
                    'MyFiles-Decrypted': false
                }
        };

    switch (originResponse.status)
    {
        case 200:
            return streamResponse (originResponse);
        case 304:
        case 307:
            const { headers } = originResponse;
            const isEncrypted = headers.get ('MyFiles-encrypted');
            if (isEncrypted)
            {
                // Fetch the file.
                let fileResponse = await fetch (headers.get ("Location"));

                switch (fileResponse.status)
                {
                    case 200:
                        let {readable, writable} = new TransformStream ();
                        let reader = fileResponse.body.getReader ({mode: "byob"});
                        let writer = writable.getWriter ();

                        decryptStream (version, firstBytes, reader, writer);

                        responseInit['headers']['MyFiles-Decrypted'] = true;
                        responseInit['headers']['content-type'] = fileResponse.headers.get ('content-type');
                        responseInit['headers']['Content-Disposition'] = fileResponse.headers.get ('Content-Disposition');

                        return new Response (readable, responseInit);
                    case 404:
                        responseInit['status'] = 404;
                        responseInit['statusText'] = "File not found";

                        return new Response ("Encrypted file not found", responseInit);
                    case 403:
                        return authFailureResponse (responseInit);
                    default:
                        responseInit['status'] = 500;
                        responseInit['statusText'] = "Error";

                        return new Response ("General Worker Failure", responseInit);
                }
            }

            return streamResponse (originResponse);
        case 403:
            return authFailureResponse ();
        case 400:
        case 404:
        case 500:
        default:
            return streamResponse (originResponse);
    }
}

I’ve stripped them down for here. Running the code works all the way up to the first time that “decryptedChunk = await crypto.subtle.decrypt (algorithm, key, cyphertextBuffer);” is called. The decryptStream () function executes up to the point of the first “await” -ed function call, then the processOriginResponse () function finishes and returns the Response object which goes all the way back to the event listener. After that, I see the code reach the loop inside of the decryptStream () function, then fall over with this error message:

decryptStream () exception while "try"-ing the "infinite" loop; ignoring.
[Error: 123145436282880:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:../deps/openssl/openssl/crypto/evp/evp_enc.c:570:

That’s the whole error output. Of course, this points to a problem with decrypting, usually something wrong with the wrong key or wrong IV, but these things I have confirmed in a few different ways (the code setting up both works in the all-in-memory test on the same encrypted file).

There are a number of supporting functions. Since they are well tested and used in the non-streaming version, I know they are working, so I didn’t include that extra code, here (just trying to keep this a shorter read). These include the generate_key_from_password (), hex2uint8Array (), string2uint8array () and base64_decode () functions.

So, where’s the flaw in my approach or code? Can subtle.crypto.decrypt () be used like this to decrypt the stream in chunks? Is there another known working approach?

You’re in luck, I’ve got some clues at least.
I dug deep into this when looking into protected video-streams.

See this thread:

While it isn’t supported in WebCrypto API yet, there’s some hacks to do this with chunks.

See this post especially:

Unfortunately, my project got cancelled so I never got the time to get working code, but it is certainly plausible. Let me know if you get it working!

Thank you for the info, @thomas4. I’ll let you know what happens.