Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ZeRO 3 Offload #834

Merged
merged 17 commits into from
Mar 8, 2021
Merged

ZeRO 3 Offload #834

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
d53ccaf
Squash stage3 v1 (#146)
jeffra Mar 3, 2021
4179dda
Fix correctness bug (#147)
jeffra Mar 3, 2021
3a2d5cd
formatting fix (#150)
Mar 4, 2021
fee2491
stage3 bugfix (API) update and simplified FP16 Z3 tests (#151)
Mar 4, 2021
fe21d21
ZeRO-3 detach and race condition bugfixes (#149)
Mar 4, 2021
4b2838f
Fix optimizer state_dict KeyError (#148)
tjruwase Mar 5, 2021
2e9025f
fix for smaller SGS sizes, ensures each grad is backed by unique tens…
jeffra Mar 5, 2021
b447a2a
Simplifying the logic for getting averaged gradients (#153)
samyam Mar 7, 2021
fb0d4fb
skip for now
jeffra Mar 8, 2021
8013615
Z3 Docs redux (#154)
Mar 8, 2021
591ca5e
removing some TODOs and commented code (#155)
Mar 8, 2021
d21a838
New Z3 defaults (#156)
Mar 8, 2021
7dec889
formatting
jeffra Mar 8, 2021
0abd60f
megatron external params
Mar 8, 2021
0f22ae1
Merge branch 'master' into staging-zero3-release
jeffra Mar 8, 2021
9efdb6b
Merge branch 'staging-zero3-release' of github.com:microsoft/DeepSpee…
Mar 8, 2021
de7e45d
Merge branch 'master' into staging-zero3-release
jeffra Mar 8, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
New Z3 defaults (#156) 
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
  • Loading branch information
@jeffra
Shaden Smith and jeffra authored Mar 8, 2021
commit d21a838b0b97e645656a81629638512fecee991d
11 changes: 8 additions & 3 deletions deepspeed/runtime/zero/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,8 @@ def _initialize(self, zero_config_dict):
self.contiguous_gradients = get_scalar_param(
zero_config_dict,
ZERO_OPTIMIZATION_CONTIGUOUS_GRADIENTS,
ZERO3_OPTIMIZATION_CONTIGUOUS_GRADIENTS_DEFAULT
if self.stage == ZERO_OPTIMIZATION_WEIGHTS else
ZERO_OPTIMIZATION_CONTIGUOUS_GRADIENTS_DEFAULT)

self.reduce_bucket_size = get_scalar_param(
Expand All @@ -95,9 +97,12 @@ def _initialize(self, zero_config_dict):
ZERO_OPTIMIZATION_REDUCE_SCATTER,
ZERO_OPTIMIZATION_REDUCE_SCATTER_DEFAULT)

self.overlap_comm = get_scalar_param(zero_config_dict,
ZERO_OPTIMIZATION_OVERLAP_COMM,
ZERO_OPTIMIZATION_OVERLAP_COMM_DEFAULT)
self.overlap_comm = get_scalar_param(
zero_config_dict,
ZERO_OPTIMIZATION_OVERLAP_COMM,
ZERO3_OPTIMIZATION_OVERLAP_COMM_DEFAULT
if self.stage == ZERO_OPTIMIZATION_WEIGHTS else
ZERO_OPTIMIZATION_OVERLAP_COMM_DEFAULT)

self.allgather_partitions = get_scalar_param(
zero_config_dict,
Expand Down
2 changes: 2 additions & 0 deletions deepspeed/runtime/zero/constants.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,9 +52,11 @@

ZERO_OPTIMIZATION_OVERLAP_COMM = 'overlap_comm'
ZERO_OPTIMIZATION_OVERLAP_COMM_DEFAULT = False
ZERO3_OPTIMIZATION_OVERLAP_COMM_DEFAULT = True

ZERO_OPTIMIZATION_CONTIGUOUS_GRADIENTS = 'contiguous_gradients'
ZERO_OPTIMIZATION_CONTIGUOUS_GRADIENTS_DEFAULT = False
ZERO3_OPTIMIZATION_CONTIGUOUS_GRADIENTS_DEFAULT = False

ZERO_OPTIMIZATION_REDUCE_BUCKET_SIZE = 'reduce_bucket_size'
ZERO_OPTIMIZATION_REDUCE_BUCKET_SIZE_DEFAULT = 500000000
Expand Down