Skip to content

Add flexibility of pipeline parallel module and engine #1399

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Oct 1, 2021
Merged

Add flexibility of pipeline parallel module and engine #1399

merged 4 commits into from
Oct 1, 2021

Conversation

@hyunwoongko
Copy link
Contributor

@hyunwoongko hyunwoongko commented Sep 25, 2021 •
edited
Loading

What for?

Add flexibility of pipeline parallel module and engine (#1347)

Request review

@ShadenSmith @jeffra @sdtblck

@hyunwoongko hyunwoongko mentioned this pull request Sep 25, 2021
Closed
@stas00
Copy link
Collaborator

stas00 commented Sep 26, 2021

Heh, @thomasw21 was just dealing with it - bigscience-workshop/Megatron-DeepSpeed#107

You can see from that PR @ShadenSmith is looking at fixing bool tensor support and removing this hack altogether.

@thomasw21
Copy link
Contributor

thomasw21 commented Sep 26, 2021 •
edited
Loading

Heh, @thomasw21 was just dealing with it - bigscience-workshop/Megatron-DeepSpeed#107

You can see from that PR @ShadenSmith is looking at fixing bool tensor support and removing this hack altogether.

Hey thanks @stas00 . Yeah I've been working to fix some issues I've been having with current pipeline parallelism implementation. I've just open a draft PR if you want to check it out. #1400 Note it doesn't work yet, and I'm in the process of debugging some deadlock issue.

@tjruwase
Copy link
Contributor

tjruwase commented Sep 27, 2021

@hyunwoongko, thanks for your hard work on this PR. I will take a look today so we can wrap it up soon. Thanks for your patience.

tjruwase
tjruwase reviewed Sep 27, 2021
View reviewed changes
tjruwase
tjruwase reviewed Sep 27, 2021
View reviewed changes
@tjruwase
Copy link
Contributor

tjruwase commented Sep 27, 2021

@hyunwoongko, I left some comments. Thanks.

@hyunwoongko hyunwoongko mentioned this pull request Sep 27, 2021
Merged
@hyunwoongko hyunwoongko changed the title Add flexibility of pipeline module and engine & fix contiguous checkpointing bugs Add flexibility of pipeline module and engine Sep 27, 2021
@hyunwoongko
Copy link
Contributor Author

hyunwoongko commented Sep 27, 2021

I have separated the two PRs separately.

@hyunwoongko hyunwoongko changed the title Add flexibility of pipeline module and engine Add flexibility of pipeline parallel module and engine Sep 28, 2021
tjruwase
tjruwase reviewed Oct 1, 2021
View reviewed changes
@tjruwase tjruwase merged commit 30965ea into deepspeedai:master Oct 1, 2021
hyunwoongko referenced this pull request in EleutherAI/DeeperSpeed Oct 2, 2021
@sdtblck
make checkpointable layers a list
2d6a0d3
@hyunwoongko hyunwoongko mentioned this pull request Nov 6, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tjruwase tjruwase tjruwase approved these changes

@awan-10 awan-10 Awaiting requested review from awan-10

@cli99 cli99 Awaiting requested review from cli99

@conglongli conglongli Awaiting requested review from conglongli

@eltonzheng eltonzheng Awaiting requested review from eltonzheng

@jeffra jeffra Awaiting requested review from jeffra

@minjiaz minjiaz Awaiting requested review from minjiaz

@niumanar niumanar Awaiting requested review from niumanar

@RezaYazdaniAminabadi RezaYazdaniAminabadi Awaiting requested review from RezaYazdaniAminabadi

@samyam samyam Awaiting requested review from samyam

@ShadenSmith ShadenSmith Awaiting requested review from ShadenSmith

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@hyunwoongko @stas00 @thomasw21 @tjruwase