FSDP tutorial (pytorch#1861)

* adding FSDP tutorial * adding DDP peak memory usage gif * adding FSDP auto wrap peak memory usage gif * adding FSDP peak memory usage gif * adding FSDP workflow image * adding FSDP tutorial * correct github handles * correct the save model mode and adding comments on availability of it in Nighlties. * correct the save model mode and adding comments on availability of it in Nighlties. * adding the image for FSDP thumbnail * adding FSDP thumbnail * correct the min parameter and model strucutre on auto _wrap * addressing comments on FSDP memory, timing and State_dict
qinwentu · Mar 17, 2022 · fe5674f · fe5674f
1 parent 07483e0
commit fe5674f
Show file tree

Hide file tree

Showing 7 changed files with 430 additions and 0 deletions.
diff --git a/_static/img/distributed/DDP_memory.gif b/_static/img/distributed/DDP_memory.gif
diff --git a/_static/img/distributed/FSDP_autowrap.gif b/_static/img/distributed/FSDP_autowrap.gif
diff --git a/_static/img/distributed/FSDP_memory.gif b/_static/img/distributed/FSDP_memory.gif
diff --git a/_static/img/distributed/fsdp_workflow.png b/_static/img/distributed/fsdp_workflow.png
diff --git a/_static/img/thumbnails/cropped/Getting-Started-with-FSDP.png b/_static/img/thumbnails/cropped/Getting-Started-with-FSDP.png
diff --git a/index.rst b/index.rst
@@ -559,6 +559,13 @@ Welcome to PyTorch Tutorials
    :link: advanced/ddp_pipeline.html
    :tags: Parallel-and-Distributed-Training
 
+.. customcarditem::
+   :header: Getting Started with Fully Sharded Data Parallel(FSDP)
+   :card_description: Learn how to train models with Fully Sharded Data Parallel package.
+   :image: _static/img/thumbnails/cropped/Getting Started with FSDP.png
+   :link: intermediate/FSDP_tutorial.html
+   :tags: Parallel-and-Distributed-Training
+
 .. Mobile
 
 .. customcarditem::

diff --git a/intermediate_source/FSDP_tutorial.rst b/intermediate_source/FSDP_tutorial.rst