Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix uneven head sequence parallelism bug (#6774) #6797

Merged
merged 11 commits into from
Dec 10, 2024

Conversation

Eugene29
Copy link
Contributor

@Eugene29 Eugene29 commented Nov 27, 2024

Here gather_idx < 2 represents is_first_all2all. During the first all2all, uneven_head_all2all will be called if either num_heads % seq_world_size != 0 or get_num_kv_heads() is None.

During the second all2all, it'll return return uneven_head_all2all if and only if get_num_kv_heads() is None which is always set during the first uneven all2all. This means that there will no longer be issue where uneven_head_all2all is returned for the second all2all because of num_heads % seq_world_size != 0.

Fixes: #6774

@Eugene29
Copy link
Contributor Author

Fix #6774

@inkcherry
Copy link
Contributor

LGTM, Many thanks!

@loadams loadams enabled auto-merge December 10, 2024 00:19
@loadams loadams disabled auto-merge December 10, 2024 23:58
@loadams loadams merged commit ecb4bf3 into microsoft:master Dec 10, 2024
10 of 11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] [Fix Suggestion] Uneven head sequence parallelism
4 participants