Can devices be specified in distributed training #3726
Description
Hello, I'm facing a problem now. I have eight gpus in one machine, and I need to optimize two models alternately in a script during training.
However, it seems that accelerate by default places both models on the same card and then copies them to all eight cards for distributed training, which leads to Out of Memory (OOM) errors during training. Is there a way in the accelerate library to specify the device where the model resides? What I want to do is to place one model on one GPU, duplicate it four times, and utilize all eight cards for distributed training.