Pass hosted image to image-to-text model

Hello,

I found a good code snippet for running the cloudflare worker ai image-to-text here but it seems the model expects an input in some kind of buffer array?

I have a hosted image url. How would I use image-to-text with a url string? Do I need to convert it somehow? (fetch the image asset then get the binary data from it somehow?)

Thanks!

You will want to fetch the image and then pass that to the AI model. Something like (untested, and needs better error handling):

const imageRes = await fetch('https://example.com/some/image.png');

// TODO: check errors, handle when it doesn't respond with 200

const bytes = await imageRes.arrayBuffer();

// TODO: might need tweaking depending on the model you're using, but `bytes` contains the raw image
const input = {
	image: bytes,
	prompt: "",
	max_tokens: 256,
};

const ai = new Ai(env.AI);

// TODO: do AI request using `input`

Thanks! I tried this but it didn’t work.

I got an error about an unexpected type. I was able to get it working by converting to Uint8Array then making an array from that like this:

	const response = await fetch(imgUrl);
	const arrayBuffer = await response.arrayBuffer();
	const arr = Array.from(new Uint8Array(arrayBuffer));

	const ai = new Ai(c.env.AI);
	const input = {
		image: arr,
		prompt: 'Provide a one sentence description of the image to be used as website alt text',
		max_tokens: 256,
	};

	const aiResponse = await ai.run<'@cf/unum/uform-gen2-qwen-500m'>('@cf/unum/uform-gen2-qwen-500m', input);
	console.log('aiREsponse:', aiResponse);
	const { description } = aiResponse;

I have to do this on the server side (my cloudflare worker), which I’m a little worried about with large images (current restriction is 4MB max).

I’m a little new to all this and just wondering how I should be thinking about my CF Worker architecture. My endpoint converts an image to bytes and then uses the image-to-text Ai on it, when should I get worried about my worker timing out? What are some smart ways to test that?

Thank you so much for the help!

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.