fix AiterFlashAttentionImpl init #20103

yuguo68 · 2025-06-25T23:30:49Z

Summary: get error TypeError: AiterFlashAttentionImpl.__init__() got multiple values for argument 'use_irope' for llama4, AiterFlashAttentionImpl.init() is missing the kv_sharing_target_layer_name arg,

vllm/vllm/attention/layer.py

Line 54 in 296ce95

kv_sharing_target_layer_name: Optional[str] = None,

Test Plan:
launch a llama4 server with this fix

Rollback Plan:

Differential Revision: D77340637

github-actions · 2025-06-25T23:30:58Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

gemini-code-assist

Summary of Changes

Hello @yuguo68, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request resolves an initialization error in the AiterFlashAttentionImpl class, which was preventing proper instantiation for certain models like Llama4. It updates the constructor signature to include a previously missing argument and clarifies the current support for KV sharing within this attention implementation.

Highlights

Bug Fix: Addresses a TypeError in AiterFlashAttentionImpl.__init__ that occurred when initializing for Llama4 models, specifically due to a missing kv_sharing_target_layer_name argument.
API Change: The kv_sharing_target_layer_name argument has been added to the __init__ method of AiterFlashAttentionImpl to align with expected parameters from calling code.
Feature Limitation: An explicit NotImplementedError is now raised if kv_sharing_target_layer_name is provided, indicating that KV sharing is not currently supported by AiterFlashAttention.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

facebook-github-bot · 2025-06-25T23:31:05Z

This pull request was exported from Phabricator. Differential Revision: D77340637

yuguo68 · 2025-06-25T23:32:34Z

cc @houseroad to review

gemini-code-assist

Code Review

The pull request adds the kv_sharing_target_layer_name argument to the AiterFlashAttentionImpl class and raises a NotImplementedError when it is used, as KV sharing is not supported in AiterFlashAttention. A suggestion was made to raise the NotImplementedError earlier in the code to improve clarity.

vllm/v1/attention/backends/rocm_aiter_fa.py

Summary: Signed-off-by: Yu Guo <yuguo@meta.com> get error ```TypeError: AiterFlashAttentionImpl.__init__() got multiple values for argument 'use_irope'``` for llama4, AiterFlashAttentionImpl.__init__() is missing the `kv_sharing_target_layer_name` arg, https://github.com/vllm-project/vllm/blob/296ce95d8e72f4c6680bda539058f48dbe0f340a/vllm/attention/layer.py#L54 Test Plan: launch a llama4 server with this fix Rollback Plan: Differential Revision: D77340637

facebook-github-bot · 2025-06-25T23:38:28Z

This pull request was exported from Phabricator. Differential Revision: D77340637

Summary: Pull Request resolved: vllm-project#20103 Signed-off-by: Yu Guo <yuguo@meta.com> get error ```TypeError: AiterFlashAttentionImpl.__init__() got multiple values for argument 'use_irope'``` for llama4, AiterFlashAttentionImpl.__init__() is missing the `kv_sharing_target_layer_name` arg, https://github.com/vllm-project/vllm/blob/296ce95d8e72f4c6680bda539058f48dbe0f340a/vllm/attention/layer.py#L54 Test Plan: launch a llama4 server with this fix Rollback Plan: Differential Revision: D77340637

Summary: Signed-off-by: Yu Guo <yuguo@meta.com> get error ```TypeError: AiterFlashAttentionImpl.__init__() got multiple values for argument 'use_irope'``` for llama4, AiterFlashAttentionImpl.__init__() is missing the `kv_sharing_target_layer_name` arg, https://github.com/vllm-project/vllm/blob/296ce95d8e72f4c6680bda539058f48dbe0f340a/vllm/attention/layer.py#L54 Test Plan: launch a llama4 server with this fix Rollback Plan: Differential Revision: D77340637

Summary: Pull Request resolved: vllm-project#20103 Signed-off-by: Yu Guo <yuguo@meta.com> get error ```TypeError: AiterFlashAttentionImpl.__init__() got multiple values for argument 'use_irope'``` for llama4, AiterFlashAttentionImpl.__init__() is missing the `kv_sharing_target_layer_name` arg, https://github.com/vllm-project/vllm/blob/296ce95d8e72f4c6680bda539058f48dbe0f340a/vllm/attention/layer.py#L54 Test Plan: launch a llama4 server with this fix Rollback Plan: Differential Revision: D77340637

facebook-github-bot · 2025-06-26T00:17:10Z

This pull request was exported from Phabricator. Differential Revision: D77340637

houseroad

Thanks for the fix!

Can you list the test plan in the PR description as well?

houseroad · 2025-06-26T00:40:59Z

vllm/v1/attention/backends/rocm_aiter_fa.py

        use_irope: bool = False,
    ) -> None:
        if blocksparse_params is not None:
            raise ValueError(
                "AiterFlashAttention does not support block-sparse attention.")
+        if kv_sharing_target_layer_name is not None:


Nit: also check it’s not empty string?

tjtanaa · 2025-06-26T01:56:18Z

@yuguo68 @houseroad This fix is also addressed in this PR #19904 . Overall, the AITER Flash Attention Backend requires a few fixes in this PR #19904.

houseroad · 2025-06-26T13:04:44Z

@yuguo68 , shall we try if #19904 fix the problem?

yuguo68 · 2025-06-26T15:44:04Z

@yuguo68 , shall we try if #19904 fix the problem?

Yes, #19904 is more comprehensive.

yuguo68 requested review from WoosukKwon, robertgshaw2-redhat, njhill, ywang96, comaniac and alexm-redhat as code owners June 25, 2025 23:30

gemini-code-assist bot reviewed Jun 25, 2025

View reviewed changes

mergify bot added the v1 label Jun 25, 2025

gemini-code-assist bot reviewed Jun 25, 2025

View reviewed changes

vllm/v1/attention/backends/rocm_aiter_fa.py Outdated Show resolved Hide resolved

yuguo68 force-pushed the export-D77340637 branch from 568cb85 to 5933173 Compare June 25, 2025 23:35

yuguo68 force-pushed the export-D77340637 branch from 5933173 to bbdde69 Compare June 25, 2025 23:38

yuguo68 force-pushed the export-D77340637 branch from bbdde69 to fa88b14 Compare June 26, 2025 00:13

yuguo68 force-pushed the export-D77340637 branch from fa88b14 to 875dcc0 Compare June 26, 2025 00:17

houseroad reviewed Jun 26, 2025

View reviewed changes

yuguo68 closed this Jun 26, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix AiterFlashAttentionImpl init #20103

fix AiterFlashAttentionImpl init #20103

yuguo68 commented Jun 25, 2025 •

edited by github-actions bot

Loading

Uh oh!

github-actions bot commented Jun 25, 2025

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

facebook-github-bot commented Jun 25, 2025

Uh oh!

yuguo68 commented Jun 25, 2025

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

Uh oh!

facebook-github-bot commented Jun 25, 2025

Uh oh!

facebook-github-bot commented Jun 26, 2025

Uh oh!

houseroad left a comment

Uh oh!

houseroad Jun 26, 2025

Uh oh!

tjtanaa commented Jun 26, 2025 •

edited

Loading

Uh oh!

houseroad commented Jun 26, 2025

Uh oh!

yuguo68 commented Jun 26, 2025

Uh oh!

Uh oh!

Uh oh!

fix AiterFlashAttentionImpl init #20103

fix AiterFlashAttentionImpl init #20103

Conversation

yuguo68 commented Jun 25, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jun 25, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Summary of Changes

Highlights

Footnotes

Uh oh!

facebook-github-bot commented Jun 25, 2025

Uh oh!

yuguo68 commented Jun 25, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

facebook-github-bot commented Jun 25, 2025

Uh oh!

facebook-github-bot commented Jun 26, 2025

Uh oh!

houseroad left a comment

Choose a reason for hiding this comment

Uh oh!

houseroad Jun 26, 2025

Choose a reason for hiding this comment

Uh oh!

tjtanaa commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

houseroad commented Jun 26, 2025

Uh oh!

yuguo68 commented Jun 26, 2025

Uh oh!

Uh oh!

yuguo68 commented Jun 25, 2025 •

edited by github-actions bot

Loading

tjtanaa commented Jun 26, 2025 •

edited

Loading