Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deepspeed activation Partitioning #18732

Open
LogicBaron opened this issue Oct 6, 2023 · 2 comments
Open

Deepspeed activation Partitioning #18732

LogicBaron opened this issue Oct 6, 2023 · 2 comments
Labels
docs Documentation related help wanted Open to be worked on strategy: deepspeed

Comments

@LogicBaron
Copy link

LogicBaron commented Oct 6, 2023

πŸ“š Documentation

Hello,

partition_activations (bool) – Enables partition activation when used with ZeRO stage 3 and model parallelism. Still requires you to wrap your forward functions in deepspeed.checkpointing.checkpoint. See deepspeed tutorial.

Upon encountering issues with activation partitioning and after checking, I found that Deepspeed activation partitioning is not significantly related to the use of zero-3; rather, it appears that the setup of model parallelism and mpu object is crucial.

Also, it is explicitly stated that pipeline parallelism, a model parallelism method provided by Deepspeed, cannot be used in conjunction with zero-2 and zero-3 from the outset.

Additionally, in the GitHub issue referenced in the official documentation, zero-stage3 and activation partitioning are used together; however, this pairing holds no particular significance.

Therefore, it is thought that there should be clearer statements regarding the use conditions for activation partitioning, beyond simply specifying that it should be used with zero3 + mp.

cc @Borda @awaelchli

@LogicBaron LogicBaron added docs Documentation related needs triage Waiting to be triaged by maintainers labels Oct 6, 2023
@CrypticRevenger
Copy link

Please assign me, I want to do it.

@awaelchli awaelchli removed the needs triage Waiting to be triaged by maintainers label Oct 6, 2023
@LogicBaron
Copy link
Author

Thank you, is there anything I need to do?

@CrypticRevenger CrypticRevenger removed their assignment Oct 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
docs Documentation related help wanted Open to be worked on strategy: deepspeed
Projects
None yet
Development

No branches or pull requests

3 participants