[New Model]: mxbai-rerank-large-v2 #20051
Description
The model to consider.
https://huggingface.co/mixedbread-ai/mxbai-rerank-large-v2
The closest model vllm already supports.
This model is built on Qwen2
What's your difficulty of supporting the model you want?
When adding the --task score argument to enable the various /rerank endpoints, I get this error:
WARNING 06-24 19:02:15 [api_server.py:816] To indicate that the rerank API is not part of the standard OpenAI API, we have located it at `/rerank`. Please update your client accordingly. (Note: Conforms to JinaAI rerank API)
INFO: 10.130.5.5:0 - "POST /v1/rerank HTTP/1.1" 500 Internal Server Error
ERROR 06-24 19:02:15 [engine.py:165] AttributeError("'Qwen2ForCausalLM' object has no attribute 'pooler'")
ERROR 06-24 19:02:15 [engine.py:165] Traceback (most recent call last):
ERROR 06-24 19:02:15 [engine.py:165] File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 163, in start
ERROR 06-24 19:02:15 [engine.py:165] self.run_engine_loop()
ERROR 06-24 19:02:15 [engine.py:165] File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 226, in run_engine_loop
ERROR 06-24 19:02:15 [engine.py:165] request_outputs = self.engine_step()
ERROR 06-24 19:02:15 [engine.py:165] ^^^^^^^^^^^^^^^^^^
ERROR 06-24 19:02:15 [engine.py:165] File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 252, in engine_step
ERROR 06-24 19:02:15 [engine.py:165] raise e
ERROR 06-24 19:02:15 [engine.py:165] File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 235, in engine_step
ERROR 06-24 19:02:15 [engine.py:165] return self.engine.step()
ERROR 06-24 19:02:15 [engine.py:165] ^^^^^^^^^^^^^^^^^^
ERROR 06-24 19:02:15 [engine.py:165] File "/usr/local/lib/python3.12/dist-packages/vllm/engine/llm_engine.py", line 1352, in step
ERROR 06-24 19:02:15 [engine.py:165] outputs = self.model_executor.execute_model(
ERROR 06-24 19:02:15 [engine.py:165] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-24 19:02:15 [engine.py:165] File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 141, in execute_model
ERROR 06-24 19:02:15 [engine.py:165] output = self.collective_rpc("execute_model",
ERROR 06-24 19:02:15 [engine.py:165] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-24 19:02:15 [engine.py:165] File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 57, in collective_rpc
ERROR 06-24 19:02:15 [engine.py:165] answer = run_method(self.driver_worker, method, args, kwargs)
ERROR 06-24 19:02:15 [engine.py:165] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-24 19:02:15 [engine.py:165] File "/usr/local/lib/python3.12/dist-packages/vllm/utils.py", line 2671, in run_method
ERROR 06-24 19:02:15 [engine.py:165] return func(*args, **kwargs)
ERROR 06-24 19:02:15 [engine.py:165] ^^^^^^^^^^^^^^^^^^^^^
ERROR 06-24 19:02:15 [engine.py:165] File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 421, in execute_model
ERROR 06-24 19:02:15 [engine.py:165] output = self.model_runner.execute_model(
ERROR 06-24 19:02:15 [engine.py:165] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-24 19:02:15 [engine.py:165] File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 06-24 19:02:15 [engine.py:165] return func(*args, **kwargs)
ERROR 06-24 19:02:15 [engine.py:165] ^^^^^^^^^^^^^^^^^^^^^
ERROR 06-24 19:02:15 [engine.py:165] File "/usr/local/lib/python3.12/dist-packages/vllm/worker/pooling_model_runner.py", line 159, in execute_model
ERROR 06-24 19:02:15 [engine.py:165] self.model.pooler(hidden_states=hidden_or_intermediate_states,
ERROR 06-24 19:02:15 [engine.py:165] ^^^^^^^^^^^^^^^^^
ERROR 06-24 19:02:15 [engine.py:165] File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1940, in __getattr__
ERROR 06-24 19:02:15 [engine.py:165] raise AttributeError(
ERROR 06-24 19:02:15 [engine.py:165] AttributeError: 'Qwen2ForCausalLM' object has no attribute 'pooler'
You can find Mixedbread's mxbai-rerank implementation for this model here.
Before submitting a new issue...
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.