Skip to content

Expert Parallelism #373

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 142 commits into
base: main
Choose a base branch
from
Open

Expert Parallelism #373

wants to merge 142 commits into from

Conversation

xrsrke
Copy link
Member

@xrsrke xrsrke commented Jun 11, 2025 •
edited
Loading

Run nanotron's moe implementation

CUDA_DEVICE_MAX_CONNECTIONS=1 torchrun --nproc_per_node=8 run_train.py --config-file  /fsx/phuc/new_workspace/snippets/experiment_configs/qwen_moe/exp20b1_3A30B_qwen_moe_and_num_experts_128_and_topk12_nanotron_imple.yaml

Run nanotron's TE implementation

/fsx/phuc/new_workspace/snippets/experiment_configs/qwen_moe/exp20b0_3A30B_qwen_moe_and_num_experts_128_and_topk12_te_imple.yaml

NouamaneTazi and others added 30 commits April 14, 2025 14:16
@NouamaneTazi
can only merge to main from dev
1ca42e3
@c8ef @NouamaneTazi
Fix UnBoundLocalError in clm_collator.py (#339)
0dbf24d 
* Update clm_collator.py

* can only merge to main from dev (#348)

---------

Co-authored-by: Nouamane Tazi <nouamane@huggingface.co>
@NouamaneTazi
InitScalingMethod
d4e9daf
@NouamaneTazi
InitScalingMethod
6e7f0fa
@NouamaneTazi
eval
24d07e5
@NouamaneTazi
try adding lightevalrunner to trainer
438257a
@NouamaneTazi
amend
4f8a350
@NouamaneTazi
amend
c9c479d
@NouamaneTazi
amend
190a6b9
@NouamaneTazi
amend
004a89c
@NouamaneTazi
amend
b4cbb55
@NouamaneTazi
amend
d39872b
@NouamaneTazi
.
feb818a
@NouamaneTazi
amend
025f314
@NouamaneTazi
amend
abe75af
@NouamaneTazi
.
bd50c66
@eliebak
qos to low
2227432
@eliebak
add nanotron_path
b62cacd
@eliebak
some fix: logs, and config
802fad6
@eliebak
cp instead of sync
895354a
@NouamaneTazi
eval_interval
55a5d3e
@NouamaneTazi
serialize sanity checks
298492e
@eliebak
add output dir and s3_save path in the config
4219ec8
@eliebak
add output dir and s3_save path in the config
f1780ec
@eliebak
fix s3 only if define
016760e
@NouamaneTazi
fixes
85138ca
@NouamaneTazi
Merge branch 'nouamane/lighteval' of https://github.com/huggingface/n…
0390de2 
…anotron into nouamane/lighteval
@eliebak
add requeue
fefb560
@xrsrke
move moe from qwen modeling to src/nn
f1160f1
@xrsrke
add groupedmlp
bb8ac96
NouamaneTazi and others added 28 commits May 12, 2025 11:33
@NouamaneTazi
Revert "rmsnorm"
83e28d5 
This reverts commit 17dad0a.
@NouamaneTazi
rope_seq_len_interpolation_factor
b4142da
@xrsrke
add moe ep
d99aa4b
@xrsrke
add saving moe checkpoints
aa4e754
@xrsrke
add resume moe checkpoints
f0d410f
@xrsrke
adapt edp to te moe
e04f6be
@xrsrke
fix edp+ep saving/resume checkpoint
d36905b
@xrsrke
use all_gather_into_tensor in token dispatcher
d24e603
@xrsrke
use fused permute, megablock's grouped gemm, and fuse multiplying exp…
559d532 
…ert output in unpermute. tested correctness with ep + edp
@NouamaneTazi
check init, fwd in case of ep=1
5d36aef
@NouamaneTazi
add all2all
df8cd20
@NouamaneTazi
loss goes down after fixing data labels
135c715
@NouamaneTazi
fix bias_activation_fusion and set to True by default
98df8cc
@NouamaneTazi
disable collect_env
88c2ee0
@NouamaneTazi
preprocess_data
acad09b
@NouamaneTazi
fix bug
1a8fb35
@NouamaneTazi
assert in case of top_k=1
d678ae7
@NouamaneTazi
add SimpleTokenDataset
e233a7d
@zzhhjjj
add timer for debug
7499256
@xrsrke
fix all-to-all
55fb95f
@xrsrke
scripts/scaling_moe_benchmark.py
aea7910
@xrsrke
fix no gradients, and expert device has no tokens
d30ae0f
@xrsrke
add compute moe params
88d4140
@xrsrke
fix sync_tied_weights_gradients without gradient accumulator
9e687c4
@xrsrke
fix sync_tied_weights_gradients
06faa91
@xrsrke
fix cannot import name 'Qwen2Config'
ce5f61c
@xrsrke
resolve merge conflicts
af69364
@xrsrke
resolve merge conflcits, fix timer in training loop
cc09277
@xrsrke xrsrke requested review from NouamaneTazi and zzhhjjj June 11, 2025 13:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@NouamaneTazi NouamaneTazi Awaiting requested review from NouamaneTazi

@zzhhjjj zzhhjjj Awaiting requested review from zzhhjjj

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@xrsrke @NouamaneTazi @c8ef @eliebak @grewalsk @zzhhjjj