[Feature]: Embedding API dimensions is currently not supported.

[Tendo33]

🚀 The feature, motivation and pitch

I noticed that after deploying the embedding model using vllm, the requests do not fully match the OpenAI format.
This issue has always been there:

{
  "object": "error",
  "message": "dimensions is currently not supported",
  "type": "BadRequestError",
  "param": null,
  "code": 400
}

Alternatives

My script:

vllm serve "/workspace/share_data/base_llms/bce-embedding-base_v1" \
    --served-model-name "bce-embedding-base_v1" \
    --task "embedding" \
    --trust-remote-code \
    --host "0.0.0.0" \
    --port 8000 \
    --dtype auto \
    --gpu-memory-utilization 0.4 \
    --kv-cache-dtype auto \
    --enable-prefix-caching \
    --tensor-parallel-size 1 \
    --max-num-seqs 256

vllm version: 0.7.2

Additional context

I hope this feature can be added as soon as possible, as it will be immensely helpful for building the knowledge base.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

Can you show how you sent the request?

[Tendo33]

Can you show how you sent the request?

After using vllm serve, the /docs of the FastAPI service, here is my Request body:

{
  "model": "bce-embedding-base_v1",
  "input": [
    "vllm"
  ],
  "encoding_format": "float",
  "dimensions": 512,
  "user": "string",
  "truncate_prompt_tokens": 1,
  "additional_data": "string",
  "add_special_tokens": true,
  "priority": 0,
  "additionalProp1": {}
}

Response body:

{
  "object": "error",
  "message": "dimensions is currently not supported",
  "type": "BadRequestError",
  "param": null,
  "code": 400
}

[DarkLight1337]

Hmm, how would dimensions work in the context of vLLM? I don't know what OpenAI is using to reduce the dimensions of their embeddings.

[Tendo33]

line
I saw this in other inference frameworks , and it seems like they just did a truncation 🤣

[DarkLight1337]

Ok, that is probably not the proper way to do it. I would rather not support it than return poor results.

[Silence-Well]

I try to pass the dimensions parameter,I encountered the same error.

INFO:     127.0.0.1:41404 - "POST /v1/embeddings HTTP/1.1" 400 Bad Request

[XYZliang]

same problem
dimensions is currently not supported

[noooop]

variable dimensions only supported by Matryoshka

https://arxiv.org/abs/2205.13147

I searched on Google
In January, OpenAI released two new embedding models: text-embedding-3-small and text-embedding-3-large. These models are trained with the Matryoshka Representation Learning technique, which lets developers trade off performance and cost in embeddings.

Sentence Transformer supports matryoshka
https://sbert.net/examples/training/matryoshka/README.html

We need to confirm that the embedding models supported by vllm are those that support the Matryoshka

[DarkLight1337]

Oh, I wasn't aware of that, thanks for pointing this out! In that case we may support dimensions for models that are trained based on this method.

[noooop]

Oh, I wasn't aware of that, thanks for pointing this out! In that case we may support dimensions for models that are trained based on this method.

I Googled it and found that none of the embedding models supported by vllm support Matryoshka. temporarily.

• BAAI/bge-base-en-v1.5 X
• BAAI/bge-multilingual-gemma2 X
• parasail-ai/GritLM-7B-vllm X
• intfloat/e5-mistral-7b-instruct X
• ssmits/Qwen2-7B-Instruct-embed-base Alibaba-NLP/gte-Qwen2-7B-instruct X
• sentence-transformers/all-roberta-large-v1 X
• intfloat/multilingual-e5-large X
• Snowflake Arctic Embed X
• jina-embeddings-v2 X

Feel free to correct me if I'm giving wrong results!

I recommend giving priority to the following models:

• tomaarsen/mpnet-base-nli-matryoshka ( mentioned in Sentence Transformer)
• jina-embeddings-v3