Support for embedding models (BGE, GTE etc)

[michaeljelly]

Embeddings models are very useful, and can easily be run on device in terms of hardware specs.

It would be awesome if swift-transformers would work with them.

[pcuenca]

Hi @michaeljelly, thanks a lot for the great suggestion! We are currently focused on making LLMs work better and faster, and I haven't really put much thought on what expanding to embedding models would entail. It could potentially be an awesome contribution from the community! Maybe @xenova and @Vaibhavs10 have more insight on what the effort could look like.

[xenova]

I actually think that embedding models are some of the easiest to add support for. 😅 Once you have BERT models supported, you automatically are able to run most of the models on the MTEB leaderboard. I see the repo already supports the BERT tokenizer, so the only additional step is to add a pooling method (typically mean or CLS pooling) to get sentence embeddings (as opposed to per-token embeddings).

Multimodal embedding models, like CLIP, can come at a later stage :)

[Vaibhavs10]

IMO, it is okay to open it to the community! I think in terms of overall impact, focusing on LLMs makes more sense since, from an On-device perspective, they'd be used way more used than an embedding model.

I believe, most of the embedding models run quite competitively on CPU already.

[pcuenca]

Thanks a lot @xenova and @Vaibhavs10 for your comments!

I assigned the good first issue label just to increase visibility, but it doesn't mean this will be a straightforward PR 😅 Would you be interested in drafting a PR @michaeljelly, if you have the time? We can guide you along the process and help where needed 😄

[Lurrobert]

any updates on this one?
@pcuenca, @michaeljelly

[pannous]

most of the embedding models run quite competitively on CPU already

sorry if I can't find this: is there anything like huggingface swift-embedders ?

[pcuenca]

Hi @pannous! Unfortunately, there's not. This is a very interesting issue that remains open for the community to explore :)

[pannous]

While not huggingface, I found this for embeddings in swift: https://github.com/ZachNagengast/similarity-search-kit
Maybe it's useful for others too and parts can be copied to huggingface? (license compatible?)

[pcuenca]

cc @ZachNagengast in case he has any feedback on his experience :)

[isbee]

For lightweight on-device embeddings, static-retrieval-mrl-en-v1 and M2V_base_glove seem like solid options. It would be great to have these integrated into the project :)

[ZachNagengast]

Also @jkrukowski put out a library that lets you skip the CoreML conversion and run the models in pure swift with MLTensor, if your app can use them: https://github.com/jkrukowski/swift-embeddings, it even loads directly from safetensors! Quite a few models in there already.

[FL33TW00D]

I'm inclined to close or convert to discussion this since @jkrukowski has provided a really good solution!
We can revisit adding embedding models directly to swift-transformers in the future :)

thoughts @pcuenca?