Fabric: drop FairScale's sharded implementation #16329

carmocca · 2023-01-10T20:45:20Z

What does this PR do?

Removes strategy='ddp_sharded'|'ddp_sharded_spawn' from LightningLite and Fabric. FSDP is the recommended replacement.

For the closest alternative, Fabric(strategy=FSDPStrategy(sharding_strategy=ShardingStrategy.SHARD_GRAD_OP)) is suggested.

Does your PR introduce any breaking changes? If yes, please list them.

Removes Fabrics and LightningLite FairScale integration

cc @Borda @justusschock @carmocca @awaelchli

github-actions · 2023-01-11T13:37:10Z

⚡ Required checks status: All passing 🟢

Groups summary

🟢 pytorch_lightning: Tests workflow

Check ID	Status
pl-cpu (macOS-11, pytorch, 3.8, 1.11)	success	✅
pl-cpu (macOS-11, pytorch, 3.9, 1.12)	success	✅
pl-cpu (macOS-11, pytorch, 3.10, 1.13)	success	✅
pl-cpu (macOS-11, pytorch, 3.8, 1.10, oldest)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.8, 1.10)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.9, 1.11)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.10, 1.12)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.10, 1.13)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.7, 1.10, oldest)	success	✅
pl-cpu (windows-2022, pytorch, 3.9, 1.11)	success	✅
pl-cpu (windows-2022, pytorch, 3.10, 1.12)	success	✅
pl-cpu (windows-2022, pytorch, 3.10, 1.13)	success	✅
pl-cpu (windows-2022, pytorch, 3.7, 1.10, oldest)	success	✅
pl-cpu (slow, macOS-11, pytorch, 3.7, 1.11)	success	✅
pl-cpu (slow, ubuntu-20.04, pytorch, 3.7, 1.11)	success	✅
pl-cpu (slow, windows-2022, pytorch, 3.7, 1.11)	success	✅
pl-cpu (macOS-11, lightning, 3.8, 1.13)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.13)	success	✅
pl-cpu (windows-2022, lightning, 3.8, 1.13)	success	✅

These checks are required after the changes to requirements/fabric/strategies.txt, src/lightning_fabric/fabric.py, src/lightning_fabric/strategies/__init__.py, src/lightning_fabric/strategies/fairscale.py, src/lightning_fabric/strategies/fsdp.py, src/lightning_fabric/strategies/strategy.py, src/pytorch_lightning/lite/lite.py, src/pytorch_lightning/overrides/fairscale.py, src/pytorch_lightning/plugins/precision/sharded_native_amp.py, src/pytorch_lightning/strategies/ddp.py, src/pytorch_lightning/strategies/fully_sharded.py, src/pytorch_lightning/strategies/sharded.py, src/pytorch_lightning/strategies/sharded_spawn.py, tests/tests_pytorch/helpers/runif.py, tests/tests_pytorch/lite/test_lite.py, tests/tests_pytorch/plugins/precision/test_sharded_precision.py, tests/tests_pytorch/strategies/test_ddp_fully_sharded_with_full_state_dict.py, tests/tests_pytorch/strategies/test_ddp_strategy.py, tests/tests_pytorch/strategies/test_sharded_strategy.py.

🟢 pytorch_lightning: Azure GPU

Check ID	Status
pytorch-lightning (GPUs)	success	✅

These checks are required after the changes to src/pytorch_lightning/lite/lite.py, src/pytorch_lightning/overrides/fairscale.py, src/pytorch_lightning/plugins/precision/sharded_native_amp.py, src/pytorch_lightning/strategies/ddp.py, src/pytorch_lightning/strategies/fully_sharded.py, src/pytorch_lightning/strategies/sharded.py, src/pytorch_lightning/strategies/sharded_spawn.py, tests/tests_pytorch/helpers/runif.py, tests/tests_pytorch/lite/test_lite.py, tests/tests_pytorch/plugins/precision/test_sharded_precision.py, tests/tests_pytorch/strategies/test_ddp_fully_sharded_with_full_state_dict.py, tests/tests_pytorch/strategies/test_ddp_strategy.py, tests/tests_pytorch/strategies/test_sharded_strategy.py, requirements/fabric/strategies.txt, src/lightning_fabric/fabric.py, src/lightning_fabric/strategies/__init__.py, src/lightning_fabric/strategies/fairscale.py, src/lightning_fabric/strategies/fsdp.py, src/lightning_fabric/strategies/strategy.py.

🟢 pytorch_lightning: Azure HPU

Check ID	Status
pytorch-lightning (HPUs)	success	✅

These checks are required after the changes to requirements/fabric/strategies.txt, src/lightning_fabric/fabric.py, src/lightning_fabric/strategies/__init__.py, src/lightning_fabric/strategies/fairscale.py, src/lightning_fabric/strategies/fsdp.py, src/lightning_fabric/strategies/strategy.py, src/pytorch_lightning/lite/lite.py, src/pytorch_lightning/overrides/fairscale.py, src/pytorch_lightning/plugins/precision/sharded_native_amp.py, src/pytorch_lightning/strategies/ddp.py, src/pytorch_lightning/strategies/fully_sharded.py, src/pytorch_lightning/strategies/sharded.py, src/pytorch_lightning/strategies/sharded_spawn.py, tests/tests_pytorch/helpers/runif.py, tests/tests_pytorch/lite/test_lite.py, tests/tests_pytorch/plugins/precision/test_sharded_precision.py, tests/tests_pytorch/strategies/test_ddp_fully_sharded_with_full_state_dict.py, tests/tests_pytorch/strategies/test_ddp_strategy.py, tests/tests_pytorch/strategies/test_sharded_strategy.py.

🟢 pytorch_lightning: Azure IPU

Check ID	Status
pytorch-lightning (IPUs)	success	✅

These checks are required after the changes to requirements/fabric/strategies.txt, src/lightning_fabric/fabric.py, src/lightning_fabric/strategies/__init__.py, src/lightning_fabric/strategies/fairscale.py, src/lightning_fabric/strategies/fsdp.py, src/lightning_fabric/strategies/strategy.py, src/pytorch_lightning/lite/lite.py, src/pytorch_lightning/overrides/fairscale.py, src/pytorch_lightning/plugins/precision/sharded_native_amp.py, src/pytorch_lightning/strategies/ddp.py, src/pytorch_lightning/strategies/fully_sharded.py, src/pytorch_lightning/strategies/sharded.py, src/pytorch_lightning/strategies/sharded_spawn.py, tests/tests_pytorch/helpers/runif.py, tests/tests_pytorch/lite/test_lite.py, tests/tests_pytorch/plugins/precision/test_sharded_precision.py, tests/tests_pytorch/strategies/test_ddp_fully_sharded_with_full_state_dict.py, tests/tests_pytorch/strategies/test_ddp_strategy.py, tests/tests_pytorch/strategies/test_sharded_strategy.py.

🟢 pytorch_lightning: Docs

Check ID	Status
make-doctest (pytorch)	success	✅
make-html (pytorch)	success	✅

These checks are required after the changes to src/pytorch_lightning/lite/lite.py, src/pytorch_lightning/overrides/fairscale.py, src/pytorch_lightning/plugins/precision/sharded_native_amp.py, src/pytorch_lightning/strategies/ddp.py, src/pytorch_lightning/strategies/fully_sharded.py, src/pytorch_lightning/strategies/sharded.py, src/pytorch_lightning/strategies/sharded_spawn.py, docs/source-pytorch/fabric/api/api_reference.rst, docs/source-pytorch/fabric/api/fabric_args.rst.

🟢 pytorch_lightning: Docker

Check ID	Status
build-cuda (3.9, 1.10, 11.3.1)	success	✅
build-cuda (3.9, 1.11, 11.3.1)	success	✅
build-cuda (3.9, 1.12, 11.6.1)	success	✅
build-cuda (3.9, 1.13, 11.7.1)	success	✅
build-hpu (1.5.0, 1.11.0)	success	✅
build-ipu (3.9, 1.10)	success	✅
build-NGC	success	✅
build-pl (3.9, 1.10, 11.3.1)	success	✅
build-pl (3.9, 1.11, 11.3.1)	success	✅
build-pl (3.9, 1.12, 11.6.1)	success	✅
build-pl (3.9, 1.13, 11.7.1)	success	✅
build-xla (3.7, 1.12)	success	✅

These checks are required after the changes to requirements/fabric/strategies.txt.

🟢 lightning_fabric: CPU workflow

Check ID	Status
fabric-cpu (macOS-11, fabric, 3.8, 1.11)	success	✅
fabric-cpu (macOS-11, fabric, 3.9, 1.12)	success	✅
fabric-cpu (macOS-11, fabric, 3.10, 1.13)	success	✅
fabric-cpu (macOS-11, fabric, 3.7, 1.10, oldest)	success	✅
fabric-cpu (ubuntu-20.04, fabric, 3.8, 1.10)	success	✅
fabric-cpu (ubuntu-20.04, fabric, 3.9, 1.11)	success	✅
fabric-cpu (ubuntu-20.04, fabric, 3.10, 1.12)	success	✅
fabric-cpu (ubuntu-20.04, fabric, 3.10, 1.13)	success	✅
fabric-cpu (ubuntu-20.04, fabric, 3.7, 1.10, oldest)	success	✅
fabric-cpu (windows-2022, fabric, 3.9, 1.11)	success	✅
fabric-cpu (windows-2022, fabric, 3.10, 1.12)	success	✅
fabric-cpu (windows-2022, fabric, 3.10, 1.13)	success	✅
fabric-cpu (windows-2022, fabric, 3.7, 1.10, oldest)	success	✅
fabric-cpu (macOS-11, lightning, 3.8, 1.13)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.8, 1.13)	success	✅
fabric-cpu (windows-2022, lightning, 3.8, 1.13)	success	✅

These checks are required after the changes to requirements/fabric/strategies.txt, src/lightning_fabric/fabric.py, src/lightning_fabric/strategies/__init__.py, src/lightning_fabric/strategies/fairscale.py, src/lightning_fabric/strategies/fsdp.py, src/lightning_fabric/strategies/strategy.py, tests/tests_fabric/helpers/models.py, tests/tests_fabric/helpers/runif.py, tests/tests_fabric/strategies/test_fairscale.py, tests/tests_fabric/strategies/test_fairscale_integration.py, tests/tests_fabric/strategies/test_fsdp_integration.py, tests/tests_fabric/strategies/test_registry.py, tests/tests_fabric/test_connector.py, tests/tests_fabric/test_fabric.py, tests/tests_fabric/utilities/test_imports.py.

🟢 lightning_fabric: Azure GPU

Check ID	Status
lightning-fabric (GPUs)	success	✅

These checks are required after the changes to requirements/fabric/strategies.txt, src/lightning_fabric/fabric.py, src/lightning_fabric/strategies/__init__.py, src/lightning_fabric/strategies/fairscale.py, src/lightning_fabric/strategies/fsdp.py, src/lightning_fabric/strategies/strategy.py, tests/tests_fabric/helpers/models.py, tests/tests_fabric/helpers/runif.py, tests/tests_fabric/strategies/test_fairscale.py, tests/tests_fabric/strategies/test_fairscale_integration.py, tests/tests_fabric/strategies/test_fsdp_integration.py, tests/tests_fabric/strategies/test_registry.py, tests/tests_fabric/test_connector.py, tests/tests_fabric/test_fabric.py, tests/tests_fabric/utilities/test_imports.py.

🟢 mypy

Check ID	Status
mypy	success	✅

These checks are required after the changes to requirements/fabric/strategies.txt, src/lightning_fabric/fabric.py, src/lightning_fabric/strategies/__init__.py, src/lightning_fabric/strategies/fairscale.py, src/lightning_fabric/strategies/fsdp.py, src/lightning_fabric/strategies/strategy.py, src/pytorch_lightning/lite/lite.py, src/pytorch_lightning/overrides/fairscale.py, src/pytorch_lightning/plugins/precision/sharded_native_amp.py, src/pytorch_lightning/strategies/ddp.py, src/pytorch_lightning/strategies/fully_sharded.py, src/pytorch_lightning/strategies/sharded.py, src/pytorch_lightning/strategies/sharded_spawn.py.

🟢 install

Check ID	Status
install-pkg (ubuntu-22.04, app, 3.7)	success	✅
install-pkg (ubuntu-22.04, app, 3.10)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.7)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.10)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.7)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.10)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.7)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.10)	success	✅
install-pkg (ubuntu-22.04, notset, 3.7)	success	✅
install-pkg (ubuntu-22.04, notset, 3.10)	success	✅
install-pkg (macOS-12, app, 3.7)	success	✅
install-pkg (macOS-12, app, 3.10)	success	✅
install-pkg (macOS-12, fabric, 3.7)	success	✅
install-pkg (macOS-12, fabric, 3.10)	success	✅
install-pkg (macOS-12, pytorch, 3.7)	success	✅
install-pkg (macOS-12, pytorch, 3.10)	success	✅
install-pkg (macOS-12, lightning, 3.7)	success	✅
install-pkg (macOS-12, lightning, 3.10)	success	✅
install-pkg (macOS-12, notset, 3.7)	success	✅
install-pkg (macOS-12, notset, 3.10)	success	✅
install-pkg (windows-2022, app, 3.7)	success	✅
install-pkg (windows-2022, app, 3.10)	success	✅
install-pkg (windows-2022, fabric, 3.7)	success	✅
install-pkg (windows-2022, fabric, 3.10)	success	✅
install-pkg (windows-2022, pytorch, 3.7)	success	✅
install-pkg (windows-2022, pytorch, 3.10)	success	✅
install-pkg (windows-2022, lightning, 3.7)	success	✅
install-pkg (windows-2022, lightning, 3.10)	success	✅
install-pkg (windows-2022, notset, 3.7)	success	✅
install-pkg (windows-2022, notset, 3.10)	success	✅

These checks are required after the changes to src/lightning_fabric/fabric.py, src/lightning_fabric/strategies/__init__.py, src/lightning_fabric/strategies/fairscale.py, src/lightning_fabric/strategies/fsdp.py, src/lightning_fabric/strategies/strategy.py, src/pytorch_lightning/lite/lite.py, src/pytorch_lightning/overrides/fairscale.py, src/pytorch_lightning/plugins/precision/sharded_native_amp.py, src/pytorch_lightning/strategies/ddp.py, src/pytorch_lightning/strategies/fully_sharded.py, src/pytorch_lightning/strategies/sharded.py, src/pytorch_lightning/strategies/sharded_spawn.py, requirements/fabric/strategies.txt.

Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

docs/source-pytorch/fabric/api/fabric_args.rst

src/lightning_fabric/CHANGELOG.md

src/pytorch_lightning/lite/lite.py

tests/tests_fabric/strategies/test_fsdp_integration.py

tests/tests_fabric/helpers/models.py

tests/tests_fabric/strategies/test_fsdp_integration.py

Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>

carmocca added deprecation Includes a deprecation breaking change Includes a breaking change fabric lightning.fabric.Fabric strategy: fairscale fsdp (removed) Fully Sharded Data Parallel labels Jan 10, 2023

carmocca added this to the v1.9 milestone Jan 10, 2023

carmocca self-assigned this Jan 10, 2023

github-actions bot added the pl Generic label for PyTorch Lightning package label Jan 10, 2023

carmocca added 3 commits January 10, 2023 23:20

Fabric: replace FairScale's sharded implementation with FSDP

f417cc8

Docstring

779a36f

Alias

900370f

carmocca force-pushed the fabric/replace-fairscale-fsdp branch from 77157f4 to 900370f Compare January 10, 2023 22:20

carmocca added 4 commits January 10, 2023 23:25

Imports

8e33df9

FSDP suppor tfor BoringFabric

b3faf2b

Simpler

754ef94

Silly pycharm

8e383fc

carmocca changed the title ~~Fabric: replace FairScale's sharded implementation with FSDP~~ Fabric: drop FairScale's sharded implementation Jan 11, 2023

carmocca added 2 commits January 11, 2023 13:49

Adrian suggests removal instead

ac016a2

Update CHANGELOG

a8f6e38

carmocca added strategy: fairscale sharded (removed) Sharded Data Parallel and removed deprecation Includes a deprecation strategy: fairscale fsdp (removed) Fully Sharded Data Parallel labels Jan 11, 2023

carmocca added 4 commits January 11, 2023 13:58

Tests

18d0598

Imports and docs

7279c09

activation checkpointing requires 1.13

35bc3f7

Run is wrapped already

df43111

carmocca marked this pull request as ready for review January 11, 2023 13:36

carmocca requested review from awaelchli, justusschock, williamFalcon and edenlightning as code owners January 11, 2023 13:36

carmocca requested review from tchaton and Borda as code owners January 11, 2023 13:36

justusschock approved these changes Jan 11, 2023

View reviewed changes

docs/source-pytorch/fabric/api/fabric_args.rst Show resolved Hide resolved

src/lightning_fabric/CHANGELOG.md Show resolved Hide resolved

awaelchli approved these changes Jan 11, 2023

View reviewed changes

src/pytorch_lightning/lite/lite.py Outdated Show resolved Hide resolved

tests/tests_fabric/strategies/test_fsdp_integration.py Show resolved Hide resolved

tests/tests_fabric/helpers/models.py Show resolved Hide resolved

mergify bot added the ready PRs ready to be merged label Jan 11, 2023

carmocca commented Jan 11, 2023

View reviewed changes

tests/tests_fabric/helpers/models.py Show resolved Hide resolved

tests/tests_fabric/strategies/test_fsdp_integration.py Outdated Show resolved Hide resolved

Apply suggestions from code review

45735d1

Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>

carmocca enabled auto-merge (squash) January 11, 2023 14:50

mergify bot added has conflicts and removed ready PRs ready to be merged labels Jan 11, 2023

Merge branch 'master' into fabric/replace-fairscale-fsdp

d899160

mergify bot added ready PRs ready to be merged and removed has conflicts ready PRs ready to be merged labels Jan 11, 2023

carmocca merged commit 428844d into master Jan 11, 2023

carmocca deleted the fabric/replace-fairscale-fsdp branch January 11, 2023 17:08

Borda mentioned this pull request Jan 12, 2023

fix import removed DDPShardedStrategy #16341

Merged

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fabric: drop FairScale's sharded implementation #16329

Fabric: drop FairScale's sharded implementation #16329

carmocca commented Jan 10, 2023 •

edited by github-actions bot

Loading

github-actions bot commented Jan 11, 2023 •

edited

Loading

Fabric: drop FairScale's sharded implementation #16329

Fabric: drop FairScale's sharded implementation #16329

Conversation

carmocca commented Jan 10, 2023 • edited by github-actions bot Loading

What does this PR do?

Does your PR introduce any breaking changes? If yes, please list them.

github-actions bot commented Jan 11, 2023 • edited Loading

⚡ Required checks status: All passing 🟢

Groups summary

carmocca commented Jan 10, 2023 •

edited by github-actions bot

Loading

github-actions bot commented Jan 11, 2023 •

edited

Loading