[Bugfix][B200] Fix cutlass_mla hang
#24966
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mergify
bot
added
documentation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a workaround to fix a hang in cutlass_mla for large batch sizes by limiting kv_splits. The fix itself seems reasonable given the context. However, I've noticed that several debugging print statements were uncommented in csrc/attention/mla/cutlass_sm100_mla/device/sm100_mla.hpp. These should be removed before merging to keep the codebase clean and avoid performance issues. The PR also includes substantial changes to dependency management files, which seem unrelated to the bugfix. It would be beneficial to address these dependency changes in a separate pull request with a dedicated description.
alexm-redhat
force-pushed
the
fix_kv_split
branch
2 times, most recently
from
bb41cec to
37f1a09
Compare
robertgshaw2-redhat
changed the title
cutlass_mla hang
mgoin
added
ready
|
In my small model tests with few prompts(BS < 8), the engine still hangs. Would it be worth investing in why there is a hang? |
|
@pavanimajety I will limit for B>1 |
alexm-redhat
force-pushed
the
fix_kv_split
branch
2 times, most recently
from
145c28f to
6870d15
Compare
…for larger batch size Signed-off-by: Alexander Matveev <amatveev@redhat.com>
alexm-redhat
force-pushed
the
fix_kv_split
branch
from
6870d15 to
2975c3f
Compare
mgoin
approved these changes
Sep 17, 2025
Thanks, that works for now. |
debroy-rh
pushed a commit
to debroy-rh/vllm
that referenced
this pull request
Sep 19, 2025
Signed-off-by: Alexander Matveev <amatveev@redhat.com> Co-authored-by: Michael Goin <mgoin64@gmail.com>
FeiDaLI
pushed a commit
to FeiDaLI/vllm
that referenced
this pull request
Sep 25, 2025
Signed-off-by: Alexander Matveev <amatveev@redhat.com> Co-authored-by: Michael Goin <mgoin64@gmail.com>
charlifu
pushed a commit
to ROCm/vllm
that referenced
this pull request
Sep 25, 2025
Signed-off-by: Alexander Matveev <amatveev@redhat.com> Co-authored-by: Michael Goin <mgoin64@gmail.com> Signed-off-by: charlifu <charlifu@amd.com>
xuebwang-amd
pushed a commit
to xuebwang-amd/vllm
that referenced
this pull request
Oct 10, 2025
Signed-off-by: Alexander Matveev <amatveev@redhat.com> Co-authored-by: Michael Goin <mgoin64@gmail.com> Signed-off-by: xuebwang-amd <xuebwang@amd.com>
choprahetarth
pushed a commit
to Tandemn-Labs/vllm
that referenced
this pull request
Oct 11, 2025
Signed-off-by: Alexander Matveev <amatveev@redhat.com> Co-authored-by: Michael Goin <mgoin64@gmail.com>
xuebwang-amd
pushed a commit
to xuebwang-amd/vllm
that referenced
this pull request
Oct 24, 2025
Signed-off-by: Alexander Matveev <amatveev@redhat.com> Co-authored-by: Michael Goin <mgoin64@gmail.com> Signed-off-by: xuebwang-amd <xuebwang@amd.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR fixes the hang issue with cutlass_mla when batch size is sufficiently large and kv_splits is high.
The solution is to limit the max kv_splits to 2 when batch size >= 1. We avoid limiting batch_size == 1, since larger kv_splits improve low-latency performance.