Hybrid Attention broken since 0.4.0 (affects Granite 4.0 H, Qwen3-Next, Jamba-3B, Nemotron-H-8b)

[Steven0236]

After upgrading vllm (compiled from source) and flashinfer, I noticed that Qwen3-Next-80b has lost a lot of precision and can't use tools and talks nonsense after the 2nd conversation turn. If I switch the backend to "FLASH_ATTN", the problem goes away. This makes me think that the problem is likely in flashinfer.

Here is the troublesome configuration:

VLLM_ATTENTION_BACKEND="FLASHINFER"
VLLM_USE_FLASHINFER_SAMPLER="0"
MAX_JOBS="16"
OMP_NUM_THREADS="16"
NVCC_THREADS="16"
PYTORCH_CUDA_ALLOC_CONF="expandable_segments:True"

vllm serve /AI2/vllm_models/Text/2509/Qwen3-Next-80B-AWQ4     \
  --trust-remote-code                                         \
  --max-model-len                  30000                      \
  --max-num-seqs                   1                          \
  --tensor-parallel-size           4                          \
  --pipeline-parallel-size         1                          \
  --enable-auto-tool-choice                                   \
  --tool-call-parser               hermes                     \
  --tokenizer-mode                 auto                       \
  --gpu-memory-utilization         0.89                       \
  --no-enable-chunked-prefill                                 

Note that I turned off flashinfer_sampler in both cases to help isolate the problem.
I also tried to downgrade flashinfer to 0.3.0 to compare, but that generated errors (maybe the latest vllm needs 0.4.0 somehow).

The troublesome versions are:

vllm:  0.11.0rc2.dev408+g55392bc87.d20251011 
Flashinfer: 0.4.0

If I go back to my earlier version setup shown below. The problem doesn't exist.

vllm:  0.11.0rc2.dev59+gfed8a9b10.d20250926
Flashinfer: 0.3.1

Any confirmation or suggestion appreciated.

PS: I will start studying the verification infrastructure and see if I can contribute some effort to help improve it later so as to catch such problems.

[Steven0236]

I've updated to 0.4.1 and the problem is still present. I noticed these errors showing up in the log:

(Worker_TP3 pid=3514) WARNING 10-15 20:08:56 [partition_rules.py:43] Failed to resolve operator for Inductor partition: vllm::mamba_mixer
(Worker_TP3 pid=3514) WARNING 10-15 20:08:56 [partition_rules.py:43] Failed to resolve operator for Inductor partition: vllm::short_conv
(Worker_TP3 pid=3514) WARNING 10-15 20:08:56 [partition_rules.py:43] Failed to resolve operator for Inductor partition: vllm::linear_attention
(Worker_TP3 pid=3514) WARNING 10-15 20:08:56 [partition_rules.py:43] Failed to resolve operator for Inductor partition: vllm::plamo2_mamba_mixer

I verified that flash-linear-attention v.0.3.2 is installed. So not sure why these warnings pop up.
Unsure if it's related to the problem.
Any help appreciated.

[yzh119]

I don't think these operators are related to flashinfer

[wenscarl]

could you show the full client script?

[Steven0236]

I see the error when running interactive applications and not a script. For example:

• using Cline and asking:
  'Read the file Oclient.py and let us discuss it.'
  it tries repeatedly to read 'src/main.js', yet there is no such folder or file on my system.
• using my personal chatbot, it responds nonsense after the second question and is unable to perform tool calls.

However, I will try to design a simple client script, tomorrow, that shows the problem so that others can try.

Trying to isolate the problem:

• Switching to flash_attn backend, everything works normally.
• Switching the model to Qwen3-4b, everything works normally.
• Switching to an older version of vllm with flashinfer 0.3.0, everything works normally.
• Unfortunately, I cannot try the latest vllm version with flashinfer 0.3.0 or the old version of vllm with flashinfer 0.4.0 because vllm is apparently locked to a given flashinfer version due to some function call parameters.

It is possible that the problem is due to some error in the way vllm now calls flashinfer 0.4.0, but it would seem a lot more likely that there is a flashinfer bug introduced in 0.4.0 that might only causes a loss of precision in the special attention mechanism of Qwen3-Next. Note that I'm running on NVIDIA sm120 GPUs,

[Steven0236]

We now have 2 people who have noticed this problem, and I've done more testing that shows the exact same problem with Granite-4.0-h-micro, Qwen3-Next-80b, and Jamba-reasoning-3b, indicating that probably all Hybrid Attention models now fail when using Flashinfer backend.

Please see vllm issue: vllm-project/vllm#26936, where I've added a simple example to test with Granite-4.0 and a simple python program that generates an openai completion API call showing the problem.
@wenscarl : Note that the client script demonstrating the problem is on the vllm issue thread.