Enable setting the sharding strategy as string in FSDP #18087

awaelchli · 2023-07-14T20:42:19Z

What does this PR do?

A minor usability update for FSDP.

Before:

from torch.distributed.fsdp import ShardingStrategy

trainer = Trainer(strategy=FSDPStrategy(sharding_strategy=ShardingStrategy.SHARD_GRAD_OP))

Now:

trainer = Trainer(strategy=FSDPStrategy(sharding_strategy="SHARD_GRAD_OP"))

cc @Borda @carmocca @justusschock @awaelchli

for more information, see https://pre-commit.ci

…tring' into feature/fsdp-shard-strategy-as-string

github-actions · 2023-07-14T22:14:16Z

⚡ Required checks status: All passing 🟢

Groups summary

🟢 pytorch_lightning: Tests workflow

Check ID	Status
pl-cpu (macOS-11, lightning, 3.8, 1.11)	success	✅
pl-cpu (macOS-11, lightning, 3.9, 1.12)	success	✅
pl-cpu (macOS-11, lightning, 3.10, 1.13)	success	✅
pl-cpu (macOS-11, lightning, 3.10, 2.0)	success	✅
pl-cpu (macOS-11, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.11)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.9, 1.12)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 1.13)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 2.0)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (windows-2022, lightning, 3.8, 1.11)	success	✅
pl-cpu (windows-2022, lightning, 3.9, 1.12)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 1.13)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 2.0)	success	✅
pl-cpu (windows-2022, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (macOS-11, pytorch, 3.8, 1.13)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.8, 1.13)	success	✅
pl-cpu (windows-2022, pytorch, 3.8, 1.13)	success	✅

These checks are required after the changes to src/lightning/fabric/strategies/fsdp.py, src/lightning/pytorch/strategies/fsdp.py, tests/tests_pytorch/strategies/test_fsdp.py.

🟢 pytorch_lightning: Azure GPU

Check ID	Status
pytorch-lightning (GPUs)	success	✅

These checks are required after the changes to src/lightning/pytorch/strategies/fsdp.py, tests/tests_pytorch/strategies/test_fsdp.py, src/lightning/fabric/strategies/fsdp.py.

🟢 pytorch_lightning: Benchmarks

Check ID	Status
lightning.Benchmarks	success	✅

These checks are required after the changes to src/lightning/fabric/strategies/fsdp.py, src/lightning/pytorch/strategies/fsdp.py.

🟢 fabric: Docs

Check ID	Status
make-doctest (fabric)	success	✅
make-html (fabric)	success	✅

These checks are required after the changes to src/lightning/fabric/strategies/fsdp.py.

🟢 pytorch_lightning: Docs

Check ID	Status
make-doctest (pytorch)	success	✅
make-html (pytorch)	success	✅

These checks are required after the changes to src/lightning/pytorch/strategies/fsdp.py.

🟢 lightning_fabric: CPU workflow

Check ID	Status
fabric-cpu (macOS-11, lightning, 3.8, 1.11)	success	✅
fabric-cpu (macOS-11, lightning, 3.9, 1.12)	success	✅
fabric-cpu (macOS-11, lightning, 3.10, 1.13)	success	✅
fabric-cpu (macOS-11, lightning, 3.10, 2.0)	success	✅
fabric-cpu (macOS-11, lightning, 3.8, 1.11, oldest)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.8, 1.11)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.9, 1.12)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.10, 1.13)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.10, 2.0)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.8, 1.11, oldest)	success	✅
fabric-cpu (windows-2022, lightning, 3.8, 1.11)	success	✅
fabric-cpu (windows-2022, lightning, 3.9, 1.12)	success	✅
fabric-cpu (windows-2022, lightning, 3.10, 1.13)	success	✅
fabric-cpu (windows-2022, lightning, 3.10, 2.0)	success	✅
fabric-cpu (windows-2022, lightning, 3.8, 1.11, oldest)	success	✅
fabric-cpu (macOS-11, fabric, 3.8, 1.13)	success	✅
fabric-cpu (ubuntu-20.04, fabric, 3.8, 1.13)	success	✅
fabric-cpu (windows-2022, fabric, 3.8, 1.13)	success	✅

These checks are required after the changes to src/lightning/fabric/strategies/fsdp.py, tests/tests_fabric/strategies/test_fsdp.py.

🟢 lightning_fabric: Azure GPU

Check ID	Status
lightning-fabric (GPUs)	success	✅

These checks are required after the changes to src/lightning/fabric/strategies/fsdp.py, tests/tests_fabric/strategies/test_fsdp.py.

🟢 mypy

Check ID	Status
mypy	success	✅

These checks are required after the changes to src/lightning/fabric/strategies/fsdp.py, src/lightning/pytorch/strategies/fsdp.py.

🟢 install

Check ID	Status
install-pkg (ubuntu-22.04, app, 3.8)	success	✅
install-pkg (ubuntu-22.04, app, 3.10)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.8)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.10)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.8)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.10)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.8)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.10)	success	✅
install-pkg (ubuntu-22.04, notset, 3.8)	success	✅
install-pkg (ubuntu-22.04, notset, 3.10)	success	✅
install-pkg (macOS-12, app, 3.8)	success	✅
install-pkg (macOS-12, app, 3.10)	success	✅
install-pkg (macOS-12, fabric, 3.8)	success	✅
install-pkg (macOS-12, fabric, 3.10)	success	✅
install-pkg (macOS-12, pytorch, 3.8)	success	✅
install-pkg (macOS-12, pytorch, 3.10)	success	✅
install-pkg (macOS-12, lightning, 3.8)	success	✅
install-pkg (macOS-12, lightning, 3.10)	success	✅
install-pkg (macOS-12, notset, 3.8)	success	✅
install-pkg (macOS-12, notset, 3.10)	success	✅
install-pkg (windows-2022, app, 3.8)	success	✅
install-pkg (windows-2022, app, 3.10)	success	✅
install-pkg (windows-2022, fabric, 3.8)	success	✅
install-pkg (windows-2022, fabric, 3.10)	success	✅
install-pkg (windows-2022, pytorch, 3.8)	success	✅
install-pkg (windows-2022, pytorch, 3.10)	success	✅
install-pkg (windows-2022, lightning, 3.8)	success	✅
install-pkg (windows-2022, lightning, 3.10)	success	✅
install-pkg (windows-2022, notset, 3.8)	success	✅
install-pkg (windows-2022, notset, 3.10)	success	✅

These checks are required after the changes to src/lightning/fabric/strategies/fsdp.py, src/lightning/pytorch/strategies/fsdp.py.

🟢 link-check

Check ID	Status
check-md-links / markdown-link-check	success	✅

These checks are required after the changes to src/lightning/fabric/CHANGELOG.md, src/lightning/pytorch/CHANGELOG.md.

Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

for more information, see https://pre-commit.ci

codecov · 2023-07-14T22:29:38Z

Codecov Report

Merging #18087 (378d88c) into master (356f5d0) will decrease coverage by 23%.
The diff coverage is 60%.

Additional details and impacted files

@@            Coverage Diff            @@
##           master   #18087     +/-   ##
=========================================
- Coverage      84%      61%    -23%     
=========================================
  Files         430      425      -5     
  Lines       32642    32559     -83     
=========================================
- Hits        27305    19799   -7506     
- Misses       5337    12760   +7423

carmocca

Cool

string

a73010c

github-actions bot added fabric lightning.fabric.Fabric pl Generic label for PyTorch Lightning package labels Jul 14, 2023

awaelchli added feature Is an improvement or enhancement strategy: fsdp Fully Sharded Data Parallel fun Staff contributions outside working hours - to differentiate from the "community" label labels Jul 14, 2023

awaelchli added this to the 2.1 milestone Jul 14, 2023

pre-commit-ci bot and others added 7 commits July 14, 2023 20:44

[pre-commit.ci] auto fixes from pre-commit.com hooks

0900b9e

for more information, see https://pre-commit.ci

import

32198cc

Merge remote-tracking branch 'origin/feature/fsdp-shard-strategy-as-s…

8722615

…tring' into feature/fsdp-shard-strategy-as-string

tests

7121859

changelog

f1ea22f

docs

423ce8a

typo

f8bb61b

awaelchli marked this pull request as ready for review July 14, 2023 21:14

awaelchli requested review from carmocca, justusschock and williamFalcon as code owners July 14, 2023 21:14

fix?

40b7283

[pre-commit.ci] auto fixes from pre-commit.com hooks

378d88c

for more information, see https://pre-commit.ci

carmocca approved these changes Jul 14, 2023

View reviewed changes

missed a spot

0db12e0

justusschock approved these changes Jul 15, 2023

View reviewed changes

mergify bot added the ready PRs ready to be merged label Jul 15, 2023

Merge branch 'master' into feature/fsdp-shard-strategy-as-string

37f5980

carmocca merged commit 080eaf3 into master Jul 15, 2023
98 checks passed

carmocca deleted the feature/fsdp-shard-strategy-as-string branch July 15, 2023 16:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable setting the sharding strategy as string in FSDP #18087

Enable setting the sharding strategy as string in FSDP #18087

awaelchli commented Jul 14, 2023 •

edited by github-actions bot

Loading

github-actions bot commented Jul 14, 2023 •

edited

Loading

codecov bot commented Jul 14, 2023 •

edited

Loading

carmocca left a comment

Enable setting the sharding strategy as string in FSDP #18087

Enable setting the sharding strategy as string in FSDP #18087

Conversation

awaelchli commented Jul 14, 2023 • edited by github-actions bot Loading

What does this PR do?

github-actions bot commented Jul 14, 2023 • edited Loading

⚡ Required checks status: All passing 🟢

Groups summary

codecov bot commented Jul 14, 2023 • edited Loading

Codecov Report

carmocca left a comment

Choose a reason for hiding this comment

awaelchli commented Jul 14, 2023 •

edited by github-actions bot

Loading

github-actions bot commented Jul 14, 2023 •

edited

Loading

codecov bot commented Jul 14, 2023 •

edited

Loading