[New Model]: Support Qwen3 Embedding & Reranker

[noooop]

Summary

• Qwen3 Embedding
  • Qwen/Qwen3-Embedding-0.6B
  • Qwen/Qwen3-Embedding-4B
  • Qwen/Qwen3-Embedding-8B
• Qwen3 Reranker
  • Qwen/Qwen3-Reranker-0.6B
  • Qwen/Qwen3-Reranker-4B
  • Qwen/Qwen3-Reranker-8B
  • tomaarsen/Qwen3-Reranker-0.6B-seq-cls

Usage

• Qwen3 Embedding

vllm serve Qwen/Qwen3-Embedding-0.6B

curl

curl http://127.0.0.1:8000/v1/embeddings \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "input": "Follow the white rabbit.",
    "model": "Qwen/Qwen3-Embedding-0.6B",
    "encoding_format": "float"
  }'

• Qwen3 Reranker

Caution

Please use the query_template and document_template to format the query and document for better reranker results.
without template, the results are almost as random. PTAL #19344

for models that have been converted into Qwen3ForSequenceClassification, such as tomaarsen/Qwen3-Reranker-0.6B-seq-cls

vllm serve tomaarsen/Qwen3-Reranker-0.6B-seq-cls

/score

curl http://127.0.0.1:8000/score \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "text_1": "ping",
    "text_2": "pong",
    "model": "tomaarsen/Qwen3-Reranker-0.6B-seq-cls"
  }'

expected output

{"id":"score-ee337e20e932467a83792d220614a7cd","object":"list","created":1749527048,"model":"tomaarsen/Qwen3-Reranker-0.6B-seq-cls","data":[{"index":0,"object":"score","score":0.06634521484375}],"usage":{"prompt_tokens":2,"total_tokens":2,"completion_tokens":0,"prompt_tokens_details":null}}

/rerank

curl http://127.0.0.1:8000/rerank \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "query": "ping",
    "documents": ["pong"],
    "model": "tomaarsen/Qwen3-Reranker-0.6B-seq-cls"
  }'

expected output

{"id":"rerank-fe06b692387444b7a56e282944f285f9","model":"tomaarsen/Qwen3-Reranker-0.6B-seq-cls","usage":{"total_tokens":2},"results":[{"index":0,"document":{"text":"pong"},"relevance_score":0.06634521484375}]}

for the official model

vllm serve Qwen/Qwen3-Reranker-0.6B --hf_overrides '{"architectures": ["Qwen3ForSequenceClassification"],"classifier_from_token": ["no", "yes"],"is_original_qwen3_reranker": true}'

Why do we need hf_overrides:
Qwen3-Reranker is a language model that doing reranker by using the logits of "no" and "yes" tokens.
vllm converts it to Qwen3ForSequenceClassification when loaded for better performance.

• Firstly, we need using "architectures": ["Qwen3ForSequenceClassification"], to manually route to Qwen3ForSequenceClassification.
• Then, we will extract the vector corresponding to classifier_from_token from lm_head using "classifier_from_token": ["no", "yes"].
• Third, we will convert these two vectors into one vector. The use of conversion logic is controlled by using "is_original_qwen3_reranker": True.

If you correctly start vllm serve, use the corresponding model name, then calling the official model is the same as the model that has been converted into Qwen3ForSequenceClassification.

/score

curl http://127.0.0.1:8000/score \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "text_1": "ping",
    "text_2": "pong",
    "model": "Qwen/Qwen3-Reranker-0.6B"
  }'

expected output

{"id":"score-7dbe101346ea4aeea4b85aa7971ddf8f","object":"list","created":1749527323,"model":"Qwen/Qwen3-Reranker-0.6B","data":[{"index":0,"object":"score","score":0.0673828125}],"usage":{"prompt_tokens":2,"total_tokens":2,"completion_tokens":0,"prompt_tokens_details":null}}

/rerank

curl http://127.0.0.1:8000/rerank \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "query": "ping",
    "documents": ["pong"],
    "model": "Qwen/Qwen3-Reranker-0.6B"
  }'

expected output

{"id":"rerank-43ddc0f96f174ae4a5eef07d51a8defd","model":"Qwen/Qwen3-Reranker-0.6B","usage":{"total_tokens":2},"results":[{"index":0,"document":{"text":"pong"},"relevance_score":0.0673828125}]}

Offline and formating query & document:

from vllm import LLM

model_name = "Qwen/Qwen3-Reranker-0.6B"

# What is the difference between the official original version and one
# that has been converted into a sequence classification model?
# Qwen3-Reranker is a language model that doing reranker by using the
# logits of "no" and "yes" tokens.
# It needs to computing 151669 tokens logits, making this method extremely
# inefficient, not to mention incompatible with the vllm score API.
# A method for converting the original model into a sequence classification
# model was proposed. See：https://huggingface.co/Qwen/Qwen3-Reranker-0.6B/discussions/3
# Models converted offline using this method can not only be more efficient
# and support the vllm score API, but also make the init parameters more
# concise, for example.
# model = LLM(model="tomaarsen/Qwen3-Reranker-0.6B-seq-cls", task="score")

# If you want to load the official original version, the init parameters are
# as follows.

model = LLM(
    model=model_name,
    task="score",
    hf_overrides={
        "architectures": ["Qwen3ForSequenceClassification"],
        "classifier_from_token": ["no", "yes"],
        "is_original_qwen3_reranker": True,
    },
)

# Why do we need hf_overrides for the official original version:
# vllm converts it to Qwen3ForSequenceClassification when loaded for
# better performance.
# - Firstly, we need using `"architectures": ["Qwen3ForSequenceClassification"],`
# to manually route to Qwen3ForSequenceClassification.
# - Then, we will extract the vector corresponding to classifier_from_token
# from lm_head using `"classifier_from_token": ["no", "yes"]`.
# - Third, we will convert these two vectors into one vector.  The use of
# conversion logic is controlled by `using "is_original_qwen3_reranker": True`.

# Please use the query_template and document_template to format the query and
# document for better reranker results.

prefix = '<|im_start|>system\nJudge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be "yes" or "no".<|im_end|>\n<|im_start|>user\n'
suffix = "<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n"

query_template = "{prefix}<Instruct>: {instruction}\n<Query>: {query}\n"
document_template = "<Document>: {doc}{suffix}"

if __name__ == "__main__":
    instruction = (
        "Given a web search query, retrieve relevant passages that answer the query"
    )

    queries = [
        "What is the capital of China?",
        "Explain gravity",
    ]

    documents = [
        "The capital of China is Beijing.",
        "Gravity is a force that attracts two bodies towards each other. It gives weight to physical objects and is responsible for the movement of planets around the sun.",
    ]

    queries = [
        query_template.format(prefix=prefix, instruction=instruction, query=query)
        for query in queries
    ]
    documents = [document_template.format(doc=doc, suffix=suffix) for doc in documents]

    outputs = model.score(queries, documents)

    print([output.outputs.score for output in outputs])

requests demo + formating query & document:

import requests

url = "http://127.0.0.1:8000/score"
MODEL_NAME = "tomaarsen/Qwen3-Reranker-0.6B-seq-cls"

# Please use the query_template and document_template to format the query and
# document for better reranker results.

prefix = '<|im_start|>system\nJudge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be "yes" or "no".<|im_end|>\n<|im_start|>user\n'
suffix = "<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n"

query_template = "{prefix}<Instruct>: {instruction}\n<Query>: {query}\n"
document_template = "<Document>: {doc}{suffix}"

instruction = (
    "Given a web search query, retrieve relevant passages that answer the query"
)

queries = [
    "What is the capital of China?",
    "Explain gravity",
]

documents = [
    "The capital of China is Beijing.",
    "Gravity is a force that attracts two bodies towards each other. It gives weight to physical objects and is responsible for the movement of planets around the sun.",
]

queries = [
    query_template.format(prefix=prefix, instruction=instruction, query=query)
    for query in queries
]
documents = [
    document_template.format(doc=doc, suffix=suffix) for doc in documents
]

response = requests.post(url,
                         json={
                             "model": MODEL_NAME,
                             "text_1": queries,
                             "text_2": documents,
                             "truncate_prompt_tokens": -1,
                         }).json()

print(response)

expected output

{'id': 'score-14f698f021b9434482ec3d94a5757e11', 'object': 'list', 'created': 1749786173, 'model': 'tomaarsen/Qwen3-Reranker-0.6B-seq-cls', 'data': [{'index': 0, 'object': 'score', 'score': 0.99951171875}, {'index': 1, 'object': 'score', 'score': 0.99951171875}], 'usage': {'prompt_tokens': 189, 'total_tokens': 189, 'completion_tokens': 0, 'prompt_tokens_details': None}}

---

Legacy

For Embedding
After merging
https://huggingface.co/Qwen/Qwen3-Embedding-0.6B/discussions/2
embeddings-benchmark/mteb#2769 (comment)
Qwen3-Embedding can already output results close to SentenceTransformers.

For Reranker

• Qwen3ForCausalLM: Qwen3 Embedding & Reranker both use the same architecture Qwen3ForCausalLM, vllm currently has no way to allow a single architecture to support Embedding and Reranker at the same time.
• SupportsCrossEncoding: For Reranker, The biggest problem is that the task Score will treat the Qwen3ForCausalLM model in a way that calculates the Score based on embedding models, which involves calculating embeddings and cosine distance. This is definitely not what was wanted.
• Qwen3ForSequenceClassification: Perhaps ultimately we need something like --hf-overrides '{"architectures": ["Qwen3ForSequenceClassification"]}' to get Qwen3 Reranker to run correctly.
• classifier_from_token: A more efficient approach is to extract token_false_id = 2152 and token_true_id = 9693 into a 2-class classification task rather than the current 151669-class classification task. We need a new interface (classifier_from_token) to implement it
• converted to a single label classification model The 2-way classifier is actually just a 1- way head classifier. https://huggingface.co/Qwen/Qwen3-Reranker-0.6B/discussions/3
• format_instruction, Should format_instruction be handled by users or vllm, and where should this piece of code be placed? I temporarily added a process_inputs callback function for LLM.score, but the online version (OpenAI-Compatible Server) doesn't know how to handle it. Please format the string using the method above.

FIX #19229
FIX #19252
FIX #19366

[gemini-code-assist]

Hello @noooop, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

Summary of Changes

Hello! Gemini here, providing a summary of this pull request. This PR aims to add initial support for Qwen3 Embedding and Reranker models within vLLM. Based on the PR title and description, the primary goal is to enable these specific Qwen3 model types. The author notes that the current implementation, particularly for the Reranker task, is a "dirty fix" due to underlying architectural challenges where both Embedding and Reranker models share the same Qwen3ForCausalLM architecture in Hugging Face, which vLLM isn't currently designed to handle cleanly for these distinct tasks. The PR focuses on implementing a specific scoring mechanism for the Reranker task by leveraging the existing causal language model structure.

Highlights

• Qwen3 Model Support: Adds initial support for Qwen3 Embedding and Reranker models.
• Reranker Scoring Logic: Implements a specific scoring method for the Reranker task by extracting logits for predefined true/false tokens (2152 and 9693) and calculating a score based on their probabilities.
• Cross-Encoding Interface: The Qwen3ForCausalLM class now implements the SupportsCrossEncoding interface, indicating its capability for tasks like reranking.
• Known Limitations: The author explicitly mentions several existing issues with this approach, including the challenge of supporting both Embedding and Reranker tasks with the same underlying architecture and the current scoring method being a workaround.

Changelog

• vllm/model_executor/models/qwen3.py
  • Added imports for pooling-related classes (LastPool, PoolingMetadata, PoolerOutput, PoolingSequenceGroupOutput) and the SupportsCrossEncoding interface.
  • Updated the Qwen3ForCausalLM class definition to inherit from SupportsCrossEncoding.
  • In the Qwen3ForCausalLM constructor, initialized a LastPool layer if the model task is set to "score".
  • Added a pooler method to Qwen3ForCausalLM which implements the custom scoring logic for the reranker task using specific token logits (2152 and 9693).

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[gemini-code-assist]

Code Review

This PR adds support for Qwen3 Embedding and Reranker models by introducing specific handling for the 'score' task within the Qwen3ForCausalLM architecture. The approach for reranking is acknowledged as a workaround and involves using logits of specific 'true'/'false' tokens. The changes are targeted and address the issues outlined in the PR description. However, the hardcoded token IDs are a concern for maintainability and generalizability.

Summary of Findings

• Hardcoded Token IDs: The pooler method in vllm/model_executor/models/qwen3.py uses hardcoded token IDs (2152 for false, 9693 for true). This is a maintainability concern and should ideally be made configurable or derived from model/tokenizer configuration.
• Clarity on Reranker Logic: The choice of LastPool in the __init__ method for the 'score' task, and how it relates to the subsequent logit-based scoring in the pooler method, could benefit from more explicit comments or documentation, especially given its characterization as a "dirty fix".

Merge Readiness

This pull request makes a good effort to support Qwen3 Reranker models within the existing Qwen3ForCausalLM architecture, acknowledging the current limitations. The approach taken is a pragmatic workaround.

However, the use of hardcoded token IDs is a significant concern that should be addressed. Ideally, these should be configurable or automatically derived. At a minimum, their origin and necessity for being hardcoded should be clearly documented in the code.

Given these points, and the author's own acknowledgement of this being a "dirty fix", I recommend addressing the hardcoded token ID issue and potentially clarifying the LastPool rationale before merging. I am unable to approve pull requests, but I suggest further discussion on these points. Other reviewers should assess the overall architectural implications of this workaround.

[noooop]

@DarkLight1337

quick review

[noooop]

tests/models/language/pooling/test_qwen3_reranker.py is a bit hastily, I am refactoring the test for score, the next PR will fix it.

[noooop]

The current code is a bit hacky.

Wait until next week to see if the official can convert the model into Qwen3ForSequenceClassification format.

[tomaarsen]

Hello!

Thank you for your useful work here @noooop. I converted the model to a sequence classification model in the meantime for me to be able to do some testing - it might come in handy for you as well: https://huggingface.co/tomaarsen/Qwen3-Reranker-0.6B-seq-cls

I can also move it to the cross-encoder organization, but I'd like to avoid that until I get models with these larger templates working more nicely with Sentence Transformers, i.e. without having to do all kinds of pre-processing manually.

• Tom Aarsen

[lovetian1991]

quick review

[zcfrank1st]

fantastic！

[NiuBlibing]

Hi, does vllm serve support instruction?

[noooop]

Hi, does vllm serve support instruction?

You can manually format the instruction when calling the API. When SentenceTransformers has an automatic instruction formatting API, we will follow up.

[NiuBlibing]

Hi, does vllm serve support instruction?

You can manually format the instruction when calling the API. When SentenceTransformers has an automatic instruction formatting API, we will follow up.

Could you provide an example by curl?

[metacryptom]

the reranker follow the above can't work

start log

INFO 06-12 12:40:17 [cli_args.py:309] non-default args: {'host': '0.0.0.0', 'model': '/aimodels/embeddigns/Qwen3-Reranker-8B', 'enforce_eager': True, 'served_model_name': ['default'], 'hf_overrides': {'architectures': ['Qwen3ForSequenceClassification'], 'classifier_from_token': ['no', 'yes'], 'is_original_qwen3_reranker': True}}
INFO 06-12 12:40:17 [config.py:533] Overriding HF config with {'architectures': ['Qwen3ForSequenceClassification'], 'classifier_from_token': ['no', 'yes'], 'is_original_qwen3_reranker': True}
INFO 06-12 12:40:23 [config.py:823] This model supports multiple tasks: {'score', 'embed', 'classify', 'generate', 'reward'}. Defaulting to 'generate'.
INFO 06-12 12:40:23 [config.py:2195] Chunked prefill is enabled with max_num_batched_tokens=2048.
WARNING 06-12 12:40:23 [cuda.py:91] To see benefits of async output processing, enable CUDA graph. Since, enforce-eager is enabled, async output processor cannot be used
WARNING 06-12 12:40:25 [env_override.py:17] NCCL_CUMEM_ENABLE is set to 0, skipping override. This may increase memory overhead with cudagraph+allreduce: NVIDIA/nccl#1234
INFO 06-12 12:40:27 [init.py:244] Automatically detected platform cuda.
INFO 06-12 12:40:28 [core.py:455] Waiting for init message from front-end.
INFO 06-12 12:40:28 [core.py:70] Initializing a V1 LLM engine (v0.9.1) with config: model='/aimodels/embeddigns/Qwen3-Reranker-8B', speculative_config=None, tokenizer='/aimodels/embeddigns/Qwen3-Reranker-8B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=40960, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=default, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=False, pooler_config=None, compilation_config={"level":0,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":[],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":0,"cudagraph_capture_sizes":[],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":0,"local_cache_dir":null}
WARNING 06-12 12:40:29 [utils.py:2737] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x7b5ac877a7e0>
INFO 06-12 12:40:30 [parallel_state.py:1065] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
INFO 06-12 12:40:30 [topk_topp_sampler.py:49] Using FlashInfer for top-p & top-k sampling.
WARNING 06-12 12:40:30 [utils.py:211] Qwen3ForSequenceClassification has no vLLM implementation, falling back to Transformers implementation. Some features may not be supported and performance may not be optimal.
INFO 06-12 12:40:30 [gpu_model_runner.py:1595] Starting to load model /aimodels/embeddigns/Qwen3-Reranker-8B...
INFO 06-12 12:40:30 [gpu_model_runner.py:1600] Loading model from scratch...
INFO 06-12 12:40:30 [transformers.py:146] Using Transformers backend.
INFO 06-12 12:40:31 [cuda.py:252] Using Flash Attention backend on V1 engine.
Loading safetensors checkpoint shards: 0% Completed | 0/5 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 20% Completed | 1/5 [00:00<00:02, 1.90it/s]
Loading safetensors checkpoint shards: 40% Completed | 2/5 [00:01<00:01, 1.82it/s]
Loading safetensors checkpoint shards: 60% Completed | 3/5 [00:01<00:00, 2.57it/s]
Loading safetensors checkpoint shards: 80% Completed | 4/5 [00:01<00:00, 2.22it/s]
Loading safetensors checkpoint shards: 100% Completed | 5/5 [00:02<00:00, 2.26it/s]
Loading safetensors checkpoint shards: 100% Completed | 5/5 [00:02<00:00, 2.21it/s]

INFO 06-12 12:40:33 [default_loader.py:272] Loading weights took 2.29 seconds
INFO 06-12 12:40:34 [gpu_model_runner.py:1624] Model loading took 15.2546 GiB and 3.626486 seconds
INFO 06-12 12:40:35 [gpu_worker.py:227] Available KV cache memory: 24.13 GiB
INFO 06-12 12:40:35 [kv_cache_utils.py:715] GPU KV cache size: 175,696 tokens
INFO 06-12 12:40:35 [kv_cache_utils.py:719] Maximum concurrency for 40,960 tokens per request: 4.29x
INFO 06-12 12:40:35 [core.py:171] init engine (profile, create kv cache, warmup model) took 1.34 seconds
INFO 06-12 12:40:35 [loggers.py:137] Engine 000: vllm cache_config_info with initialization after num_gpu_blocks is: 10981
WARNING 06-12 12:40:35 [config.py:1363] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with --generation-config vllm.
INFO 06-12 12:40:35 [serving_chat.py:118] Using default chat sampling params from model: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
INFO 06-12 12:40:35 [serving_completion.py:66] Using default completion sampling params from model: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
INFO 06-12 12:40:35 [api_server.py:1349] Starting vLLM API server 0 on http://0.0.0.0:8000
INFO 06-12 12:40:35 [launcher.py:29] Available routes are:
INFO 06-12 12:40:35 [launcher.py:37] Route: /openapi.json, Methods: HEAD, GET
INFO 06-12 12:40:35 [launcher.py:37] Route: /docs, Methods: HEAD, GET
INFO 06-12 12:40:35 [launcher.py:37] Route: /docs/oauth2-redirect, Methods: HEAD, GET
INFO 06-12 12:40:35 [launcher.py:37] Route: /redoc, Methods: HEAD, GET
INFO 06-12 12:40:35 [launcher.py:37] Route: /health, Methods: GET
INFO 06-12 12:40:35 [launcher.py:37] Route: /load, Methods: GET
INFO 06-12 12:40:35 [launcher.py:37] Route: /ping, Methods: POST
INFO 06-12 12:40:35 [launcher.py:37] Route: /ping, Methods: GET
INFO 06-12 12:40:35 [launcher.py:37] Route: /tokenize, Methods: POST
INFO 06-12 12:40:35 [launcher.py:37] Route: /detokenize, Methods: POST
INFO 06-12 12:40:35 [launcher.py:37] Route: /v1/models, Methods: GET
INFO 06-12 12:40:35 [launcher.py:37] Route: /version, Methods: GET
INFO 06-12 12:40:35 [launcher.py:37] Route: /v1/chat/completions, Methods: POST
INFO 06-12 12:40:35 [launcher.py:37] Route: /v1/completions, Methods: POST
INFO 06-12 12:40:35 [launcher.py:37] Route: /v1/embeddings, Methods: POST
INFO 06-12 12:40:35 [launcher.py:37] Route: /pooling, Methods: POST
INFO 06-12 12:40:35 [launcher.py:37] Route: /classify, Methods: POST
INFO 06-12 12:40:35 [launcher.py:37] Route: /score, Methods: POST
INFO 06-12 12:40:35 [launcher.py:37] Route: /v1/score, Methods: POST
INFO 06-12 12:40:35 [launcher.py:37] Route: /v1/audio/transcriptions, Methods: POST
INFO 06-12 12:40:35 [launcher.py:37] Route: /rerank, Methods: POST
INFO 06-12 12:40:35 [launcher.py:37] Route: /v1/rerank, Methods: POST
INFO 06-12 12:40:35 [launcher.py:37] Route: /v2/rerank, Methods: POST
INFO 06-12 12:40:35 [launcher.py:37] Route: /invocations, Methods: POST
INFO 06-12 12:40:35 [launcher.py:37] Route: /metrics, Methods: GET
INFO: Started server process [1]
INFO: Waiting for application startup.
INFO: Application startup complete.

when I run curl http://127.0.0.1:9211/rerank -H 'accept: application/json' -H 'Content-Type: application/json' -d '{
"query": "ping",
"documents": ["pong"],
"model": "default"
}'

It got
{"object":"error","message":"The model does not support Rerank (Score) API","type":"BadRequestError","param":null,"code":400}

when execute curl http://127.0.0.1:9211/score
-H 'accept: application/json'
-H 'Content-Type: application/json'
-d '{
"text_1": "ping",
"text_2": "pong",
"model": "default"
}'

It got
{"object":"error","message":"The model does not support Score API","type":"BadRequestError","param":null,"code":400}r

[noooop]

the reranker follow the above can't work

This pr didn't catch up the 0.9.1 release, so you need to install the dev version.

https://docs.vllm.ai/en/latest/getting_started/installation/gpu.html#install-the-latest-code_1

[noooop]

Could you provide an example by curl?

Perhaps I should add a request demo code.

done

[TPLink32]

按照上述方法重新排序无法工作

此 pr 没有赶上 0.9.1 版本，因此您需要安装开发版本。

https://docs.vllm.ai/en/latest/getting_started/installation/gpu.html#install-the-latest-code_1

按照上述方法重新排序无法工作

此 pr 没有赶上 0.9.1 版本，因此您需要安装开发版本。

https://docs.vllm.ai/en/latest/getting_started/installation/gpu.html#install-the-latest-code_1

pip show vllm
Name: vllm
Version: 0.9.2.dev96+gf40f763f1
Summary: A high-throughput and memory-efficient inference and serving engine for LLMs
Home-page: https://github.com/vllm-project/vllm
Author: vLLM Team

{"object":"error","message":"The model does not support Embeddings API","type":"BadRequestError","param":null,"code":400}

[noooop]

按照上述方法重新排序无法工作

此 pr 没有赶上 0.9.1 版本，因此您需要安装开发版本。
https://docs.vllm.ai/en/latest/getting_started/installation/gpu.html#install-the-latest-code_1

按照上述方法重新排序无法工作

此 pr 没有赶上 0.9.1 版本，因此您需要安装开发版本。
https://docs.vllm.ai/en/latest/getting_started/installation/gpu.html#install-the-latest-code_1

pip show vllm Name: vllm Version: 0.9.2.dev96+gf40f763f1 Summary: A high-throughput and memory-efficient inference and serving engine for LLMs Home-page: https://github.com/vllm-project/vllm Author: vLLM Team

{"object":"error","message":"The model does not support Embeddings API","type":"BadRequestError","param":null,"code":400}

try

uv pip install -U vllm --extra-index-url https://wheels.vllm.ai/nightly --torch-backend=auto

[TPLink32]

NFO 06-17 11:00:04 [init.py:244] Automatically detected platform cuda.
INFO 06-17 11:00:08 [api_server.py:1287] vLLM API server version 0.9.2.dev110+g119f68394
INFO 06-17 11:00:08 [cli_args.py:309] non-default args: {'host': '0.0.0.0', 'port': 7778, 'model': 'Qwen3-Reranker-0.6B-seq-cls', 'max_model_len': 12000, 'served_model_name': ['rer'], 'gpu_memory_utilization': 0.4, 'max_num_seqs': 8}
INFO 06-17 11:00:16 [config.py:831] This model supports multiple tasks: {'embed', 'classify', 'score', 'reward'}. Defaulting to 'score'.
INFO 06-17 11:00:16 [config.py:3270] Downcasting torch.float32 to torch.float16.
INFO 06-17 11:00:16 [config.py:1444] Using max model len 12000
WARNING 06-17 11:00:16 [arg_utils.py:1665] --task score is not supported by the V1 Engine. Falling back to V0.
INFO 06-17 11:00:16 [api_server.py:265] Started engine process with PID 3851273
INFO 06-17 11:00:19 [init.py:244] Automatically detected platform cuda.
INFO 06-17 11:00:22 [llm_engine.py:230] Initializing a V0 LLM engine (v0.9.2.dev110+g119f68394) with config: model='Qwen3-Reranker-0.6B-seq-cls', speculative_config=None, tokenizer='Qwen3-Reranker-0.6B-seq-cls', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=12000, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=rer, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=None, chunked_prefill_enabled=False, use_async_output_proc=False, pooler_config=PoolerConfig(pooling_type=None, normalize=None, softmax=None, step_tag_id=None, returned_token_ids=None), compilation_config={"level":0,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":[],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":0,"cudagraph_capture_sizes":[8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":8,"local_cache_dir":null}, use_cached_outputs=True,
INFO 06-17 11:00:24 [cuda.py:336] Using Flash Attention backend.
[W617 11:00:34.947629619 socket.cpp:200] [c10d] The hostname of the client socket cannot be retrieved. err=-3
[W617 11:00:44.958527474 socket.cpp:200] [c10d] The hostname of the client socket cannot be retrieved. err=-3
INFO 06-17 11:00:44 [parallel_state.py:1072] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
INFO 06-17 11:00:44 [model_runner.py:1171] Starting to load model Qwen3-Reranker-0.6B-seq-cls...

tomaarsen/Qwen3-Reranker-0.6B-seq-cls work
curl http://127.0.0.1:7778/rerank -H 'accept: application/json' -H 'Content-Type: application/json' -d '{
"query": "ping",
"documents": ["pong"],
"model": "rer"
}'
{"id":"rerank-18af6afa320b490681ec6daecdac909a","model":"rer","usage":{"total_tokens":2},"results":[{"index":0,"document":{"text":"pong"},"relevance_score":0.06632687151432037}]}

Qwen/Qwen3-Reranker-0.6B
{"object":"error","message":"The model does not support Rerank (Score) API","type":"BadRequestError","param":null,"code":400}

[CallmeZhangChenchen]

@noooop
Hi! the speed of Qwen3-Embedding-0.6B using vllm is very slow

GPU 4090

---

vllm:0.9.1

vllm serve /models/Qwen3-Embedding-0.6B --trust-remote-code

begin = time.time()
    responses = client.embeddings.create(
        input=["Hi! Follow the white rabbit."],
        model=model,
    )
    print(time.time() - begin)

100ms

---

sentence_transformers

model = SentenceTransformer("/models/Qwen3-Embedding-0.6B")
tensor = model.encode("Hi! Follow the white rabbit.")

33ms

[noooop]

@noooop Hi! the speed of Qwen3-Embedding-0.6B using vllm is very slow

GPU 4090

vllm:0.9.1

vllm serve /models/Qwen3-Embedding-0.6B --trust-remote-code

begin = time.time()
    responses = client.embeddings.create(
        input=["Hi! Follow the white rabbit."],
        model=model,
    )
    print(time.time() - begin)

100ms

sentence_transformers

model = SentenceTransformer("/models/Qwen3-Embedding-0.6B")
tensor = model.encode("Hi! Follow the white rabbit.")

33ms

Using batch operations might be faster,
vllm's advantage is in throughput rather than latency

[noooop]

Qwen/Qwen3-Reranker-0.6B {"object":"error","message":"The model does not support Rerank (Score) API","type":"BadRequestError","param":null,"code":400}

Have you try the command below?

for the official model

vllm serve Qwen/Qwen3-Reranker-0.6B --hf_overrides '{"architectures": ["Qwen3ForSequenceClassification"],"classifier_from_token": ["no", "yes"],"is_original_qwen3_reranker": true}'

[noob-ctrl]

@noooop It seems not support tp_size > 1

[noooop]

@noooop It seems not support tp_size > 1

I can't test tp>1 temporarily, welcome to contributing.

[TPLink32]

en3ForSequenceClassification"],"cla

thinks
New problem with embedding
INFO 06-17 16:37:49 [engine.py:317] Added request embd-be953f7ccae14a96b5f9f8e13b16b4d7-61.
INFO 06-17 16:37:49 [engine.py:317] Added request embd-be953f7ccae14a96b5f9f8e13b16b4d7-62.
INFO 06-17 16:37:49 [engine.py:317] Added request embd-be953f7ccae14a96b5f9f8e13b16b4d7-63.
ERROR 06-17 16:37:49 [engine.py:165] AttributeError("'PlaceholderBlockSpaceManager' object has no attribute 'remove_seq_from_computed_blocks_tracker'")
ERROR 06-17 16:37:49 [engine.py:165] Traceback (most recent call last):
ERROR 06-17 16:37:49 [engine.py:165] File "/data/code/llm_test/vllm_env/lib/python3.11/site-packages/vllm/engine/multiprocessing/engine.py", line 163, in start
ERROR 06-17 16:37:49 [engine.py:165] self.run_engine_loop()
ERROR 06-17 16:37:49 [engine.py:165] File "/data/code/llm_test/vllm_env/lib/python3.11/site-packages/vllm/engine/multiprocessing/engine.py", line 226, in run_engine_loop
ERROR 06-17 16:37:49 [engine.py:165] request_outputs = self.engine_step()
ERROR 06-17 16:37:49 [engine.py:165] ^^^^^^^^^^^^^^^^^^
ERROR 06-17 16:37:49 [engine.py:165] File "/data/code/llm_test/vllm_env/lib/python3.11/site-packages/vllm/engine/multiprocessing/engine.py", line 252, in engine_step
ERROR 06-17 16:37:49 [engine.py:165] raise e
ERROR 06-17 16:37:49 [engine.py:165] File "/data/code/llm_test/vllm_env/lib/python3.11/site-packages/vllm/engine/multiprocessing/engine.py", line 235, in engine_step
ERROR 06-17 16:37:49 [engine.py:165] return self.engine.step()
ERROR 06-17 16:37:49 [engine.py:165] ^^^^^^^^^^^^^^^^^^
ERROR 06-17 16:37:49 [engine.py:165] File "/data/code/llm_test/vllm_env/lib/python3.11/site-packages/vllm/engine/llm_engine.py", line 1296, in step
ERROR 06-17 16:37:49 [engine.py:165] ) = self.scheduler[virtual_engine].schedule()
ERROR 06-17 16:37:49 [engine.py:165] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-17 16:37:49 [engine.py:165] File "/data/code/llm_test/vllm_env/lib/python3.11/site-packages/vllm/core/scheduler.py", line 1553, in schedule
ERROR 06-17 16:37:49 [engine.py:165] scheduler_outputs: SchedulerOutputs = self._schedule()
ERROR 06-17 16:37:49 [engine.py:165] ^^^^^^^^^^^^^^^^
ERROR 06-17 16:37:49 [engine.py:165] File "/data/code/llm_test/vllm_env/lib/python3.11/site-packages/vllm/core/scheduler.py", line 1512, in _schedule
ERROR 06-17 16:37:49 [engine.py:165] return self._schedule_default()
ERROR 06-17 16:37:49 [engine.py:165] ^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-17 16:37:49 [engine.py:165] File "/data/code/llm_test/vllm_env/lib/python3.11/site-packages/vllm/core/scheduler.py", line 1277, in _schedule_default
ERROR 06-17 16:37:49 [engine.py:165] prefills = self._schedule_prefills(budget,
ERROR 06-17 16:37:49 [engine.py:165] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-17 16:37:49 [engine.py:165] File "/data/code/llm_test/vllm_env/lib/python3.11/site-packages/vllm/core/scheduler.py", line 1195, in _schedule_prefills
ERROR 06-17 16:37:49 [engine.py:165] self.remove_seq_from_computed_blocks_tracker(
ERROR 06-17 16:37:49 [engine.py:165] File "/data/code/llm_test/vllm_env/lib/python3.11/site-packages/vllm/core/scheduler.py", line 1715, in remove_seq_from_computed_blocks_tracker
ERROR 06-17 16:37:49 [engine.py:165] self._remove_seq_from_computed_blocks_tracker(seq)
ERROR 06-17 16:37:49 [engine.py:165] File "/data/code/llm_test/vllm_env/lib/python3.11/site-packages/vllm/core/scheduler.py", line 1722, in _remove_seq_from_computed_blocks_tracker
ERROR 06-17 16:37:49 [engine.py:165] self.block_manager.remove_seq_from_computed_blocks_tracker(seq)
ERROR 06-17 16:37:49 [engine.py:165] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-17 16:37:49 [engine.py:165] AttributeError: 'PlaceholderBlockSpaceManager' object has no attribute 'remove_seq_from_computed_blocks_tracker'

[noooop]

ERROR 06-17 16:37:49 [engine.py:165] AttributeError: 'PlaceholderBlockSpaceManager' object has no attribute 'remove_seq_from_computed_blocks_tracker'

fixed by #19686

[dengcao]

You can use Vllm to deploy the Qwen3-Reranker large model.There is already a temporary solution:
ModelsCope：
https://www.modelscope.cn/models/dengcao/Qwen3-Reranker-8B
https://www.modelscope.cn/models/dengcao/Qwen3-Reranker-4B
https://www.modelscope.cn/models/dengcao/Qwen3-Reranker-0.6B

GitHub：
https://github.com/dengcao/Qwen3-Reranker-8B
https://github.com/dengcao/Qwen3-Reranker-4B
https://github.com/dengcao/Qwen3-Reranker-0.6B