Skip to content

Commit 5f0556a

Browse files
subramenSvetlana Karslioglu
andauthored
Add dist_tuto to resources (#2317)
Co-authored-by: Svetlana Karslioglu <svekars@fb.com>
1 parent 5de40c6 commit 5f0556a

File tree

1 file changed

+3
-0
lines changed

1 file changed

+3
-0
lines changed

beginner_source/ddp_series_theory.rst

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,8 @@ ensures each device gets a non-overlapping input batch. The model is replicated
3737
each replica calculates gradients and simultaneously synchronizes with the others using the `ring all-reduce
3838
algorithm <https://tech.preferred.jp/en/blog/technologies-behind-distributed-deep-learning-allreduce/>`__.
3939

40+
This `illustrative tutorial <https://pytorch.org/tutorials/intermediate/dist_tuto.html#>`__ provides a more in-depth python view of the mechanics of DDP.
41+
4042
Why you should prefer DDP over DataParallel (DP)
4143
-------------------------------------------------
4244

@@ -66,3 +68,4 @@ Further Reading
6668
API <https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html>`__
6769
- `DDP Internal
6870
Design <https://pytorch.org/docs/master/notes/ddp.html#internal-design>`__
71+
- `DDP Mechanics Tutorial <https://pytorch.org/tutorials/intermediate/dist_tuto.html#>`__

0 commit comments

Comments
 (0)