It could be nice to have an analog to train_ddp.py for LocalSGD (train_localsgd.py), although the README does reference local_sgd_integ_test.py as an example.
I have the code written and would love to open a PR if people feel like this is valuable.
Potentially opening up an examples directory to host these scripts would be valuable?