Batching external call to external service

Developers Workers
1

Hi there,

I am working on a small worker that logs to an external service (Kinesis firehose) and then does a redirect

I am trying to batch the external calls together to avoid hitting ingestion limits. Is this the correct way to do it or is there a better way? (p.s i looked at using queues but the cost would be high for what we need)

Thank you

Josh

UPDATE: this seems to work locally but not when i deploy using wrangler

import { createUrl, getFinalUrl, getUserId, isValidUrl } from "./lib/url";
import { sendToFirehose } from "./lib/sendToFirehose";

export interface Env {
  AWS_ACCESS_KEY_ID: string;
  AWS_SECRET_ACCESS_KEY: string;
}

let batch: Record<any, any>[] = [];
export default {
  async fetch(request: Request, env: Env): Promise<Response> {
    const { pathname, searchParams } = new URL(request.url);
    ////
    
    const data = {
      date: new Date(),
      url: request.url,  
    };

    batch.push(data);

    if (batch.length === 100) {
      console.log("Batch size reached, sending to firehose");
      await sendToFirehose(batch);
      batch = [];
    }

    return Response.redirect("http://newurl.com", 302);
  },
};
2

Queues is going to be the best way to batch because otherwise you would need the worker to run the entire time the batch is being generated

3

The isolate can be removed before the 100 is hit, you’ll still want to send in those cases. You can do that with a waitUntil, I do this myself in some services.

Sample code:

import { createUrl, getFinalUrl, getUserId, isValidUrl } from "./lib/url";
import { sendToFirehose } from "./lib/sendToFirehose";

export interface Env {
  AWS_ACCESS_KEY_ID: string;
  AWS_SECRET_ACCESS_KEY: string;
}

let batch: Record<any, any>[] | null = null;
export default {
  async fetch(request: Request, env: Env, ctx: ExecutionContext): Promise<Response> {
    const { pathname, searchParams } = new URL(request.url);
    ////
    
    const data = {
      date: new Date(),
      url: request.url,  
    };

    if (batch === null) {
      // We have a new batch, make the array and queue
      batch = [];
      ctx.waitUntil(new Promise(async (resolve) => {
        if (batch.length > 0) {
          await sendToHirehose(batch);
          batch = null;
        }
      }));
    } else if (batch.length === 100) {
      console.log("Batch size reached, sending to firehose");
      await sendToFirehose(batch);
      batch = null;
    } else {
      batch.push(data);
    }

    return Response.redirect("http://newurl.com", 302);
  },
};
4

in the end i did go with queues, i make a over calculation the first time when working out costs