Best practices for multi-GPU training?

[gkaissis]

Congratulations on the new release! Could you maybe provide official recommendations for how to train on multiple GPUs, possibly alongside a full example? The examples in the repo fail due to unset environment variables and I am not sure which integration (Data Parallel, Distributed Data Parallel etc.) to use. The official PyTorch documentation is very thorough but not exactly intuitive for someone just wanting to run a model quickly. My use-case is that I would like to train a large dataset on a single machine with two or more GPUs with opacus. This, I believe, is what most users would like to do so I believe an end-to-end tutorial would be very useful.

I also tried Lightning, but this fails when setting >1 GPU. Is this something which is anticipated or am I doing something wrong. Your official example doesn’t include multi-GPU support AFAIK.

Thank you very much!

[ashkan-software]

Hi @gkaissis

Thank you for using Opacus and bringing this up to our attention!

I will mark this issue as an improvement to our documentation, as I also think have a standalone tutorial for multi-GPU training could benefit our users. Specially a use-case like yours (train a large dataset on a single machine with two or more GPUs with Opacus). I will also discuss this internally with the team :)

In the meantime, I am including 2 of our examples that actually support multi-GPU training:

• https://github.com/pytorch/opacus/blob/main/examples/vision_benchmark.py
• https://github.com/pytorch/opacus/blob/main/examples/cifar10.py (for this, see the PR: Multi-GPU example on CIFAR-10 #170)

I hope this helps!

[Yuancheng-Xu]

I wonder what is the difference between DDP and DPDDP. My use case is that I only need the per-sample gradient functionality. Thank you!

[alexandresablayrolles]

@Yuancheng-Xu: DPDDP is our version of DistributedDataParallel that includes DP aspects (i.e., clipping). The TL;DR is to use DPDDP when you do flat clipping and DDP when you do per-layer clipping, see L287-294 of https://github.com/pytorch/opacus/blob/main/examples/cifar10.py#L287.

@gkaissis: The way to run on multiple GPUs is similar to the regular Pytorch way, e.g. for the CIFAR example it is

python -m torch.distributed.launch --nproc_per_node 2 examples/cifar10.py

[ffuuugor]

The official tutorial is now available at https://github.com/pytorch/opacus/blob/main/tutorials/ddp_tutorial.ipynb

Closing this issue, feel free to reopen or report a new one related to the tutorial