Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Multiple zero stage 3 related fixes #3886

Merged
merged 28 commits into from
Jul 28, 2023
Merged

Multiple zero stage 3 related fixes #3886

merged 28 commits into from
Jul 28, 2023

Conversation

tjruwase
Copy link
Contributor

@tjruwase tjruwase commented Jul 5, 2023 •
edited
Loading

  • zero.Init composition
  • Add missing tensor allocators: torch.randn, torch.tensor
  • Remove assert in overridden module.apply
  • module.apply override disabling option
    @stas00

TODO

  • module.apply override documentation
  • [ ]

tjruwase added 18 commits July 5, 2023 19:17
@tjruwase
Merge branch 'master' into olruwase/override_module_apply
262b57d
@tjruwase
Removing early partitioning in override
0920c0c
@tjruwase
Merge branch 'olruwase/override_module_apply' of github.com:microsoft…
854e76e 
…/DeepSpeed into olruwase/override_module_apply
@tjruwase
Merge branch 'master' into olruwase/override_module_apply
2a04012
@tjruwase
Merge branch 'master' into olruwase/override_module_apply
5f21888
@tjruwase
Unit tests
c4fdf77
@tjruwase
Merge branch 'olruwase/override_module_apply' of github.com:microsoft…
cfe0f57 
…/DeepSpeed into olruwase/override_module_apply
@tjruwase
Merge branch 'master' into olruwase/override_module_apply
8de71db
@tjruwase
Cleanup
7c088b3
@tjruwase
Merge branch 'olruwase/override_module_apply' of github.com:microsoft…
297485f 
…/DeepSpeed into olruwase/override_module_apply
@tjruwase
Merge branch 'master' into olruwase/override_module_apply
4fd1f3c
@tjruwase
Merge branch 'master' into olruwase/override_module_apply
972c958
@tjruwase
Merge branch 'master' into olruwase/override_module_apply
0f5a508
@tjruwase
Adapt unit test to succeed
2343cfe
@tjruwase
Merge branch 'master' into olruwase/override_module_apply
7a0c339
@tjruwase
Handle missed params
fcb3bad
@tjruwase
Merge branch 'olruwase/override_module_apply' of github.com:microsoft…
dd6f334 
…/DeepSpeed into olruwase/override_module_apply
@tjruwase
Merge branch 'master' into olruwase/override_module_apply
afa4a03
@tjruwase tjruwase requested a review from tohtana July 25, 2023 22:06
tohtana
tohtana reviewed Jul 27, 2023
View reviewed changes
Copy link
Contributor

@tohtana tohtana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall this looks good to me. I left a brief comment.

deepspeed/runtime/zero/partition_parameters.py Show resolved Hide resolved
@tjruwase tjruwase added this pull request to the merge queue Jul 28, 2023
Merged via the queue into master with commit 7f90ef4 Jul 28, 2023
@tjruwase tjruwase mentioned this pull request Aug 2, 2023
Merged
polisettyvarma pushed a commit to polisettyvarma/DeepSpeed that referenced this pull request Aug 7, 2023
@tjruwase @polisettyvarma
Multiple zero stage 3 related fixes (microsoft#3886)
9ac00b0 
* Option to override module apply

* Removing early partitioning in override

* Unit tests

* Cleanup

* Adapt unit test to succeed

* Handle missed params

* Add accelerate

* Code cleanup

* Add doc

* Add doc

* Add doc
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tohtana tohtana tohtana approved these changes

@jeffra jeffra Awaiting requested review from jeffra

@samyam samyam Awaiting requested review from samyam

@mrwyattii mrwyattii Awaiting requested review from mrwyattii

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@tjruwase @tohtana