mxbai-rerank-large-v2

[qdrddr]

Model description

Currently mixedbread-ai/mxbai-rerank-large-v2 does not support rerank. Reason: the loaded model cannot fullfill rerank. Options are 'embed'.

infinity_emb v2 --port 7997 \
  --model-id mixedbread-ai/mxbai-rerank-large-v2 --batch-size 8 --revision "refs/heads/main"

curl --location 'http://localhost:7997/v1/rerank' \
--header 'Accept: application/json' \
--header 'Content-Type: application/json' \
--data '{
    "model": "mixedbread-ai/mxbai-rerank-large-v2",
    "query": "What is the capital of the United States?",
    "documents": [
        "Carson City is the capital city of the American state of Nevada.",
        "The Commonwealth of the Northern Mariana Islands is a group of islands in the Pacific Ocean. Its capital is Saipan.",
        "Washington, D.C. is the capital of the United States.",
        "Capital punishment has existed in the United States since before it was a country."
    ],
    "top_n": 3
  }'

{"error":{"message":"ModelNotDeployedError: model=`mixedbread-ai/mxbai-rerank-large-v2` does not support `rerank`. Reason: the loaded moded cannot fullyfill `rerank`. Options are {'embed'}.","type":null,"param":null,"code":400}}

Open source status & huggingface transformers.

• The model implementation is available on transformers
• The model weights are available on huggingface-hub
• I verified that the model is currently not running in the latest version pip install infinity_emb[all] --upgrade
• I made the authors of the model aware that I want to use it with infinity_emb & check if they are aware of the issue.

[michaelfeil]

https://huggingface.co/michaelfeil/mxbai-rerank-large-v2-seq
https://huggingface.co/michaelfeil/mxbai-rerank-base-v2-seq

The prompt for classification and background in this script: https://github.com/michaelfeil/infinity/blob/main/docs/lm_head_to_classifier/convert_lm.py

I'd recommend running this model as a "classifier" and applying the chat template similar to the mixedbread v2 reference implementation locally (client side!)

[qdrddr]

Something like this?
https://github.com/qdrddr/infinity-mxbai-rerank-seq-v2

[michaelfeil]

Your prompt is incorrect, use this once, together with chat template.
/rerank is not going to cut it, it needs to be a sequence classification / classify endpoint model.

https://github.com/mixedbread-ai/mxbai-rerank/blob/ca0c55d03770d9bb183ca759850bf7cdfbcc9f50/mxbai_rerank/mxbai_rerank_v2.py#L34

Edit: I added the template here: https://huggingface.co/michaelfeil/mxbai-rerank-large-v2-seq

[qdrddr]

Thanks, that worked!