Question: Support for colbertv2.0 ?

[shatealaboxiaowang]

Model description

Hi dear:
Thanks for your source code. can support for colbertv2.0 deployment ?

Thank you!

Open source status

• The model implementation is available on transformers
• The model weights are available on huggingface-hub
• I verified that the model is currently not running in the lastest version pip install infinity_emb[all] --upgrade

Provide useful links for the implementation

No response

[michaelfeil]

Colbert is a late-interaction model (stateful).

Please provide some example code with only torch and the transformers library. I think it requires some client side computation (late interaction). Don't use any third party packages like colbert package.

[wirthual]

Hi @shatealaboxiaowang ,

You are able to run colbertv2 with infinity like so:

infinity_emb v2 --model-id colbert-ir/colbertv2.0

[simjak]

@wirthual can you run it on RunPod?

[michaelfeil]

@simjak same as Colleen, you can’t use infinity serverless yet. You can spin up your own serverfull Runpod etc.

[simjak]

@michaelfeil any plans to support serveless ColPali?
Is there an example of spinning off ColPali pod?

[michaelfeil]

port=7997
model1=michaelfeil/colqwen2-v0.1
model2=colbert-ir/colbertv2.0

# needs 16GB+
docker run -it --gpus all \
 -p $port:$port \
 michaelf34/infinity:latest \
 v2 \
 --model-id $model1 \
 --model-id $model2 \
 --port $port \
--dtype bfloat16 \
--batch-size 8 \
--device cuda

[simjak]

@michaelfeil I tried to run on runpod, but got:

2025-01-03T13:14:45.429197971Z huggingface_hub.errors.EntryNotFoundError: 404 Client Error. (Request ID: Root=1-6777e2c5-19b02a382e0c73337abbfc1f;bd37d90f-3a39-4744-a1bd-c70bb380dfba)
2025-01-03T13:14:45.429203568Z Entry Not Found for url: https://huggingface.co/vidore/colqwen2-v1.0/resolve/main/config.json.
2025-01-03T13:14:45.429208543Z ERROR:    Application startup failed. Exiting.

Is there something wrong with this model https://huggingface.co/vidore/colqwen2-v1.0

[simjak]

oh, I needed to use the merged version https://huggingface.co/vidore/colqwen2-v1.0-merged