[Feature]: Support Qwen3 Embedding & Reranker

[ccdv-ai]

🚀 The feature, motivation and pitch

Core architecture is Qwen3 (0.6B, 4B, 8B).

The reranker relies on a specific logic to rank (softmax between logits of "yes" and "no" tokens):
https://huggingface.co/Qwen/Qwen3-Reranker-0.6B

The embedding model is more standard:
https://huggingface.co/Qwen/Qwen3-Embedding-0.6B

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[sigridjineth]

Look at this one for embedding usage.

# Requires vllm>=0.8.5
# https://github.com/QwenLM/Qwen3-Embedding?tab=readme-ov-file#vllm-usage
import logging
from typing import Dict, Optional, List

import json
import logging

import torch

from transformers import AutoTokenizer, is_torch_npu_available
from vllm import LLM, SamplingParams
from vllm.distributed.parallel_state import destroy_model_parallel
import gc
import math
from vllm.inputs.data import TokensPrompt

def format_instruction(instruction, query, doc):
    text = [
        {"role": "system", "content": "Judge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be \"yes\" or \"no\"."},
        {"role": "user", "content": f"<Instruct>: {instruction}\n\n<Query>: {query}\n\n<Document>: {doc}"}
    ]
    return text

def process_inputs(pairs, instruction, max_length, suffix_tokens):
    messages = [format_instruction(instruction, query, doc) for query, doc in pairs]
    messages =  tokenizer.apply_chat_template(
        messages, tokenize=True, add_generation_prompt=False, enable_thinking=False
    )
    messages = [ele[:max_length] + suffix_tokens for ele in messages]
    messages = [TokensPrompt(prompt_token_ids=ele) for ele in messages]
    return messages

def compute_logits(model, messages, sampling_params, true_token, false_token):
    outputs = model.generate(messages, sampling_params, use_tqdm=False)
    scores = []
    for i in range(len(outputs)):
        final_logits = outputs[i].outputs[0].logprobs[-1]
        token_count = len(outputs[i].outputs[0].token_ids)
        if true_token not in final_logits:
            true_logit = -10
        else:
            true_logit = final_logits[true_token].logprob
        if false_token not in final_logits:
            false_logit = -10
        else:
            false_logit = final_logits[false_token].logprob
        true_score = math.exp(true_logit)
        false_score = math.exp(false_logit)
        score = true_score / (true_score + false_score)
        scores.append(score)
    return scores

number_of_gpu = torch.cuda.device_count()
tokenizer = AutoTokenizer.from_pretrained('Qwen/Qwen3-Reranking-4B')
model = LLM(model='Qwen/Qwen3-Reranking-0.6B', tensor_parallel_size=number_of_gpu, max_model_len=10000, enable_prefix_caching=True, gpu_memory_utilization=0.8)
tokenizer.padding_side = "left"
tokenizer.pad_token = tokenizer.eos_token
suffix = "<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n"
max_length=8192
suffix_tokens = tokenizer.encode(suffix, add_special_tokens=False)
true_token = tokenizer("yes", add_special_tokens=False).input_ids[0]
false_token = tokenizer("no", add_special_tokens=False).input_ids[0]
sampling_params = SamplingParams(temperature=0, 
    max_tokens=1,
    logprobs=20, 
    allowed_token_ids=[true_token, false_token],
)

        
task = 'Given a web search query, retrieve relevant passages that answer the query'
queries = ["What is the capital of China?",
    "Explain gravity",
]
documents = [
    "The capital of China is Beijing.",
    "Gravity is a force that attracts two bodies towards each other. It gives weight to physical objects and is responsible for the movement of planets around the sun.",
]

pairs = list(zip(queries, documents))
inputs = process_inputs(pairs, task, max_length-len(suffix_tokens), suffix_tokens)
scores = compute_logits(model, inputs, sampling_params, true_token, false_token)
print('scores', scores)

destroy_model_parallel()

[czhcc]

vllm=0.8.5
Qwen3-Embedding可以通过/v1/embeddings调用
Qwen3-Reranker不行，用/v1/rerank调用返回The model does not support Rerank (Score) API

[noooop]

Qwen3-Embedding needs this to get the correct result.
https://huggingface.co/Qwen/Qwen3-Embedding-0.6B/discussions/2
embeddings-benchmark/mteb#2769 (comment)

[lovetian1991]

vllm=0.8.5 Qwen3-Embedding可以通过/v1/embeddings调用 Qwen3-Reranker不行，用/v1/rerank调用返回The model does not support Rerank (Score) API

这个会不会考虑兼容下 Rerank (Score) API

[sinopec]

embedding works in 0.9.0.1 while set task=embed, rerank failed because of " 'Qwen3ForCausalLM' object has no attribute 'pooler'"

[noooop]

embedding works in 0.9.0.1 while set task=embed, rerank failed because of " 'Qwen3ForCausalLM' object has no attribute 'pooler'"

working on it

[ccdv-ai]

The reranker model can be converted to a single label classification model:
https://huggingface.co/Qwen/Qwen3-Reranker-0.6B/discussions/3

[SeiunSky0131]

vllm=0.8.5 Qwen3-Embedding可以通过/v1/embeddings调用 Qwen3-Reranker不行，用/v1/rerank调用返回The model does not support Rerank (Score) API

Could you give an example about how to call /v1/embeddings using curl command? I tried several different methods but found all of them failed. I do use vllm=0.8.5. Many thanks!

[44670]

for qwen3 ranker models, a batched API that prefills the prompt and then returns the output logits of selected tokens (yes/no) would be helpful.

do we have such an api for now?

[xiakj]

mark

[vaclcer]

is the ranker function really fixed? I am still getting

'Qwen3ForCausalLM' object has no attribute 'pooler'

error on v0.9.1

edit:

ohh, I see... #19260 (comment)