[BugFix][FlashInfer] Fix attention backend interface mismatch with unexpected keyword use_irope

[YUNQIUGUO]

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results

Purpose

Fix the interface mismatch for FlashInfer attention backend for use_irope.

Error:

(VllmWorker rank=5 pid=2407942) ERROR 06-03 14:44:28 [multiproc_executor.py:486]   File "/home/guorachel/venv/vllm/vllm/model_executor/models/llama4.py", line 262, in __init__
(VllmWorker rank=5 pid=2407942) ERROR 06-03 14:44:28 [multiproc_executor.py:486]     self.self_attn = Llama4Attention(
(VllmWorker rank=5 pid=2407942) ERROR 06-03 14:44:28 [multiproc_executor.py:486]                      ^^^^^^^^^^^^^^^^
(VllmWorker rank=5 pid=2407942) ERROR 06-03 14:44:28 [multiproc_executor.py:486]   File "/home/guorachel/venv/vllm/vllm/model_executor/models/llama4.py", line 195, in __init__
(VllmWorker rank=5 pid=2407942) ERROR 06-03 14:44:28 [multiproc_executor.py:486]     self.attn = Attention(
(VllmWorker rank=5 pid=2407942) ERROR 06-03 14:44:28 [multiproc_executor.py:486]                 ^^^^^^^^^^
(VllmWorker rank=5 pid=2407942) ERROR 06-03 14:44:28 [multiproc_executor.py:486]   File "/home/guorachel/venv/vllm/vllm/attention/layer.py", line 134, in __init__
(VllmWorker rank=5 pid=2407942) ERROR 06-03 14:44:28 [multiproc_executor.py:486]     self.impl = impl_cls(num_heads, head_size, scale, num_kv_heads,
(VllmWorker rank=5 pid=2407942) ERROR 06-03 14:44:28 [multiproc_executor.py:486]                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=5 pid=2407942) ERROR 06-03 14:44:28 [multiproc_executor.py:486] TypeError: FlashInferImpl.__init__() got an unexpected keyword argument 'use_irope'

Test Plan

VLLM_USE_V1=1 VLLM_ATTENTION_BACKEND=FLASHINFER vllm serve meta-llama/Llama-4-Scout-17B-16E-Instruct --disable-log-requests -tp 8 --max-num-seqs 64 --no-enable-prefix-caching --max_num_batched_tokens=81920 --max-model-len=32768

Accuracy eval:

FLASHINFER_ENABLE_AOT=1 VLLM_USE_V1=1 VLLM_ATTENTION_BACKEND=FLASHINFER lm_eval --model vllm --model_args "pretrained=meta-llama/Llama-4-Scout-17B-16E-Instruct,tensor_parallel_size=8,max_model_len=32768" --trust_remote_code --tasks gsm8k --num_fewshot 5 --batch_size auto

Flashinfer on H100 (with this diff:)

FlashInfer:

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.9196	±	0.0075
		strict-match	5	exact_match	↑	0.9052	±	0.0081

Flashinfer on B200 (with this diff:)

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.9219	±	0.0074
		strict-match	5	exact_match	↑	0.9037	±	0.0081

Test Result

After this interface fix, the server command starts successfully.

[gemini-code-assist]

Hello @YUNQIUGUO, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

Summary of Changes

Hello! Gemini here, providing a summary of this pull request to help everyone get up to speed quickly.

This PR addresses a TypeError that occurs when initializing the FlashInfer attention backend. The error message indicates that the FlashInferImpl.__init__ method received an unexpected keyword argument, specifically use_irope. This suggests an interface mismatch where the calling code (likely the main Attention layer) is passing use_irope, but the FlashInfer implementation's constructor doesn't accept it.

The core change involves modifying the constructors in both vllm/attention/layer.py and vllm/v1/attention/backends/flashinfer.py to explicitly accept the use_irope boolean parameter and store it as an instance attribute. This aligns the interface and resolves the TypeError, allowing the vLLM server to start successfully with the FlashInfer backend enabled, as demonstrated by the provided test plan and result.

Highlights

• Bug Fix: Fixes a TypeError when initializing the FlashInfer attention backend due to an unexpected use_irope keyword argument.
• Interface Alignment: Modifies the __init__ methods of the main attention layer and the FlashInfer implementation to accept the use_irope parameter.
• FlashInfer Backend: Ensures compatibility for the use_irope parameter specifically within the FlashInfer attention implementation.

Changelog

• vllm/attention/layer.py
  • Added use_irope: bool = False parameter to the __init__ method signature.
  • Stored the use_irope parameter as an instance attribute self.use_irope.
• vllm/v1/attention/backends/flashinfer.py
  • Added use_irope: bool = False parameter to the FlashInferImpl.__init__ method signature.
  • Stored the use_irope parameter as an instance attribute self.use_irope.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

---

A keyword unexpected,
A type error detected.
Interface mismatch,
A quick code patch,
Now the backend's perfected.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[gemini-code-assist]

Code Review

This pull request effectively addresses the TypeError encountered when initializing the FlashInfer attention backend with the use_irope keyword argument. The changes correctly add the missing parameter to both the abstract Attention layer and the concrete FlashInferImpl.

The code is clean and directly fixes the reported bug. However, I noticed that the use_irope parameter is added to the FlashInferImpl but doesn't appear to be used within the implementation. This raises a question about its intended purpose in this specific backend.

Summary of Findings

• Unused use_irope parameter in FlashInferImpl: The use_irope parameter is added to the FlashInferImpl constructor and stored, but it is not used in the class's logic. Clarification or implementation is needed.

Merge Readiness

The core issue causing the TypeError is resolved by this pull request. The changes are minimal and directly target the problem. The only notable point is the unused use_irope parameter in the FlashInfer implementation. While not critical for the bug fix itself, clarifying its purpose or implementing its functionality (if intended) would improve the code's maintainability and correctness. I recommend addressing the comment regarding the unused parameter before merging. I am unable to approve this pull request; please have other reviewers approve this code before merging.

[YUNQIUGUO]

cc @chenyang78

[houseroad]

Thanks for the fix. Could you also check the accuracy? Like gsm8k, etc.

[YUNQIUGUO]

Thanks for the fix. Could you also check the accuracy? Like gsm8k, etc.

Sorry for the late reply.

Here's FlashInfer backend gsm8k:

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.0114	±	0.0029
		strict-match	5	exact_match	↑	0.0114	±	0.0029

As a reference, FlashAttn backend gsm8k:

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.9189	±	0.0075
		strict-match	5	exact_match	↑	0.9014	±	0.0082

let me know if this is enough/looks okay.

[YUNQIUGUO]

force-push'ed to overwrite with non signed off commit..

[mgoin]

The FlashInfer gsm8k results look very poor.. I think there is still an issue

[chenyang78]

The FlashInfer gsm8k results look very poor.. I think there is still an issue

Thanks, Michael! This PR tries to fix a compilation error related to the FlashInfer backend. I don't recall that we tested the Maverick fp8 model with FlashInfer, so it's quite likely the issue already exists. We will look into it.

[YUNQIUGUO]

The FlashInfer gsm8k results look very poor.. I think there is still an issue

yeah agreed the numbers look wrong... as another reference, also tried running with v0 engine for flashinfer:

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.4458	±	0.0137
		strict-match	5	exact_match	↑	0.4382	±	0.0137

the accuracy is higher but still quite poor.

[YUNQIUGUO]

The FlashInfer gsm8k results look very poor.. I think there is still an issue

also the commands I am running:

FLASHINFER_ENABLE_AOT=1 VLLM_USE_V1=1 VLLM_ATTENTION_BACKEND=FLASHINFER lm_eval --model vllm --model_args "pretrained=meta-llama/Llama-4-Scout-17B-16E-Instruct,tensor_parallel_size=8,max_model_len=32768" --trust_remote_code --tasks gsm8k --num_fewshot 5 --batch_size auto

similarly, set VLLM_USE_V1=0 for v0 engine.

[houseroad]

Can we also run another one with different backend, like FlashAttention, to confirm the accuracy gap?

[chenyang78]

Can we also run another one with different backend, like FlashAttention, to confirm the accuracy gap?

Yeah, @YUNQIUGUO I think we can probably try FlashAttention on V0 engine

[YUNQIUGUO]

yeah, re @houseroad @chenyang78

for flash_attn v1:

refer to the second table in my first comment: #19134 (comment)

for flash_attn v0:

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.9174	±	0.0076
		strict-match	5	exact_match	↑	0.8999	±	0.0083

seems like there's no accuracy gap for flashattn backend between v0 v.s. v1

[chenyang78]

Thanks, Rachel! Looks like FlashInfer backend does have some numerical issue for the Maverick 128E model

[YUNQIUGUO]

any more things we need to confirm here? or maybe would it be possible to let this PR go in to fix the compilation error first?

Thanks!

[chenyang78]

@YUNQIUGUO What's your flash-infer version and CUDA version? I am able to get good eval numbers with this PR on both B200 and H100

My versions:

nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Fri_Jan__6_16:45:21_PST_2023
Cuda compilation tools, release 12.0, V12.0.140
Build cuda_12.0.r12.0/compiler.32267302_0

flash infer

commit a023006a0a16143fffc938ad6b44b5d6f345cf04 (HEAD -> main, origin/main, origin/HEAD)
Author: eigen <52445717+yyihuang@users.noreply.github.com>
Date:   Sun Jun 8 22:29:10 2025 -0700

vllm:

commit 12e582922145bc7441e2f20e451dd161d4ffe828 (HEAD -> llama-breakdown)
Author: Reid <61492567+reidliu41@users.noreply.github.com>
Date:   Mon Jun 9 15:26:12 2025 +0800

B200 results:

$ VLLM_USE_V1=1 VLLM_ATTENTION_BACKEND=FLASHINFER lm_eval --model vllm --model_args "pretrained=meta-llama/Llama-4-Scout-17B-16E-Instruct,tensor_parallel_size=8,max_model_len=32768" --trust_remote_code --tasks gsm8k --num_fewshot 5 --batch_size auto
...
2025-06-09:02:39:24,306 INFO     [lm_eval.loggers.evaluation_tracker:272] Output path not provided, skipping saving results aggregated
vllm (pretrained=meta-llama/Llama-4-Scout-17B-16E-Instruct,tensor_parallel_size=8,max_model_len=32768,trust_remote_code=True), gen_kwargs: (None), limit: None, num_fewshot: 5, batch_size: auto
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.9219|±  |0.0074|
|     |       |strict-match    |     5|exact_match|↑  |0.9037|±  |0.0081|

H100 results:

$ VLLM_USE_V1=1 VLLM_ATTENTION_BACKEND=FLASHINFER lm_eval --model vllm --model_args "pretrained=meta-llama/Llama-4-Scout-17B-16E-Instruct,tensor_parallel_size=8,max_model_len=32768" --trust_remote_code --tasks gsm8k --num_fewshot 5 --batch_size auto
(VllmWorker rank=0 pid=1874535) INFO 06-09 03:14:06 [cuda.py:227] Using FlashInfer backend on V1 engine.                                                  (VllmWorker rank=7 pid=1874591) WARNING 06-09 03:14:06 [flashinfer.py:514] Using irope in FlashInfer is not supported yet, it will fall back to global att
ention for long context.                                                                                                                                  
...
vllm (pretrained=meta-llama/Llama-4-Scout-17B-16E-Instruct,tensor_parallel_size=8,max_model_len=32768,trust_remote_code=True), gen_kwargs: (None), limit: None, num_fewshot: 5, batch_size: auto
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.9242|±  |0.0073|
|     |       |strict-match    |     5|exact_match|↑  |0.9075|±  |0.0080|

Could you double-check?

[YUNQIUGUO]

re @chenyang78:

Thanks so much for double-checking the lm eval results on B200 and H100! it's good to know that the eval results are good. (which should be expected that this PR only intend to fix the compilation error :) )

And sorry for the confusion, I guess my vllm version is fairly old. let me update to latest main and retry

[YUNQIUGUO]

Thanks @chenyang78!

using vllm 0.8.5.dev1214+g8058c9110 (latest main commit version)

I am able to repro good gsm8k numbers too: (H100) (v1 engine)

FlashInfer:

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.9196	±	0.0075
		strict-match	5	exact_match	↑	0.9052	±	0.0081

[YUNQIUGUO]

@mgoin Hi Michael! I think @chenyang78 and I double checked the accuracy eval should be all good now, could you please kindly give another eye and review if no other concern?

Thanks in advance!

[chenyang78]

Great! Thanks for confirming this, @YUNQIUGUO .

cc @houseroad

[houseroad]

Thanks for the fix.

[houseroad]

Could you update the test plan to include the accuracy eval results?

[mgoin]

Glad there is no problem :)