[Feature request] Separate text & image embeddings

[josephrocca]

This is already on your radar:

embedding (either text or image, not necessarily both) is a use-case which should probably be supported. I'll look into it :)

• [Question] Using CLIP for simple image-text similarity #136

but I ran into the need again and I figured I'd create an issue for this -- if only so people (including myself) can subscribe to updates on this feature.

Edit: Oh also, regarding this comment from that issue:

you can technically get that done with zero-shot image classification, but your use-case might mean you'll need to do the similarity checks yourself

I'm fine with doing the cosine similarity stuff myself - how would one use the zero-shot image classification stuff to just get text and image embeddings individually? Is that currently possible at all?

[xenova]

I'm fine with doing the cosine similarity stuff myself - how would one use the zero-shot image classification stuff to just get text and image embeddings individually? Is that currently possible at all?

I believe this is a limitation in how the models are exported to ONNX with optimum, where both image and text inputs are required. I would need to look into it a bit more, but if this is indeed the case, it would probably be best to open a feature request in optimum.

[josephrocca]

(Came back to check on this and realised I didn't create an issue in the Optimum repo - just created one)

[xenova]

Thanks! 😇