[Question] Using CLIP for simple image-text similarity

[josephrocca]

I'm trying to get a simple image-text similarity thing working with CLIP, and I'm not sure how to do it, or whether it's currently supported with Transformers.js outside of the zero-shot image classification pipeline.

Is there a code example somewhere to get me started? Here's what I have so far:

import { AutoModel, AutoTokenizer } from 'https://cdn.jsdelivr.net/npm/@xenova/transformers@2.1.1';
let tokenizer = await AutoTokenizer.from_pretrained('Xenova/clip-vit-base-patch16');
let model = await AutoModel.from_pretrained('Xenova/clip-vit-base-patch16');
let inputIds = await tokenizer(["cat", "astronaut"]);
let image = await fetch("https://i.imgur.com/fYhUGoY.jpg").then(r => r.blob());
// how to process the image, and how to pass the image and inputIds to `model`?

Here's what I see if I inspect the model function in DevTools:

I also tried this:

import { AutoModel, AutoTokenizer, AutoProcessor } from 'https://cdn.jsdelivr.net/npm/@xenova/transformers@2.1.1';
let model = await AutoModel.from_pretrained('Xenova/clip-vit-base-patch16');
let processor = await AutoProcessor.from_pretrained("Xenova/clip-vit-base-patch16");
let inputs = await processor({text:["a photo of a cat", "a photo of an astronaut"], images:["https://i.imgur.com/fYhUGoY.jpg"]});
let outputs = await model(inputs);

But it seems that processor expects an array of images, or something? The above code throws an error saying that an .rgb() method should exist on the input.

[xenova]

Sure thing. Here's some example code. The main thing is the use of RawImage class to help with loading the images.

import { AutoModel, AutoProcessor, AutoTokenizer, RawImage, env } from '@xenova/transformers';

env.allowLocalModels = false;

const modelID = 'Xenova/clip-vit-base-patch16';

// Load tokenizer, processor, and model
let tokenizer = await AutoTokenizer.from_pretrained(modelID);
let processor = await AutoProcessor.from_pretrained(modelID);
let model = await AutoModel.from_pretrained(modelID);

// Run tokenization
let texts = [
    'a photo of a car',
    'a photo of a football match',
]
let text_inputs = tokenizer(texts, {
    padding: true,
    truncation: true
});

// Read image and run processor
let image = await RawImage.read('https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/football-match.jpg');
let image_inputs = await processor(image);

// Run model with both text and pixel inputs
let output = await model({ ...text_inputs, ...image_inputs });
console.log(output);

which outputs:

{
    "logits_per_image": {
        "dims": [ 1, 2 ],
        "type": "float32",
        "data": [ 19.406269073486328, 25.282390594482422 ],
		"size": 2
    },
    "logits_per_text": {
        "dims": [ 2, 1 ],
        "type": "float32",
        "data": [ 19.406269073486328, 25.282390594482422 ],
        "size": 2
    },
    "text_embeds": {
        "dims": [ 2, 512 ],
        "type": "float32",
        "data": [ ... ],
        "size": 1024
    },
    "image_embeds": {
        "dims": [ 1, 512 ],
        "type": "float32",
        "data": [ ... ],
        "size": 512
    }
}

---

In general, if you can use pipeline, I'd recommend using it since it's easier to keep the output standardized (i.e., the same as the python library). The rest may be relatively experimental... so, if you do want to use the auto classes, you may need to update your code when a later version comes out. That said, I'll add some more docs for RawImage and some of the other classes.

[josephrocca]

Perfect, thank you!

if you can use pipeline, I'd recommend using it since it's easier to keep the output standardized

Just to be clear - there's no pipeline for image-text similarity yet, right? Unless feature-extraction is what I was looking for here?

I'll add some more docs for RawImage and some of the other classes.

👍👍👍 and especially some simple example snippets for common scenarios and popular models like the one you provided here. Docs that include lots of example snippets are a life-saver for me - helps me wrap my head around the abstractions, and after looking a couple of examples of related tasks it's usually much more clear where in the API reference I should look to learn more about my specific task.

Edit: Is model({ ...text_inputs }) supposed to work? I.e. just getting text embeddings - no images. It gives an error saying that pixel_values is required.

[xenova]

Just to be clear - there's no pipeline for image-text similarity yet, right?

Well, you can technically get that done with zero-shot image classification, but your use-case might mean you'll need to do the similarity checks yourself.

Is model({ ...text_inputs }) supposed to work? I.e. just getting text embeddings - no images. It gives an error saying that pixel_values is required.

The model is exported with pixel_values as a required input, but I see that just embedding (either text or image, not necessarily both) is a use-case which should probably be supported. I'll look into it :)