Specify GPUs bug (torch.distributed.all_reduce(torch.zeros(1).cuda()))

[MM-IR]

On my server, when the cuda:0 is full, then I face with this issue each time, when I specify other GPus with CUDA_VISIBLE_DEVICES..

[MM-IR]

Oh, I find that they are still taking the first two GPUs by ray::worker when I specify other two.

[MM-IR]

I also find that when running multiple distributed inferences (processes) on a single server, with multiple GPUs, if the first ray one mainly uses the cuda:0 and cuda:1, then the next process would take cuda:2 and cuda:3 given the full uses of existing case, w/o following any "CUDA_VISIBLE_DEVICES".

My situation: I also use hydra, I am not sure which one causes such issues.

[MM-IR]

Or is it impossible to enable multiple ray instances simultaneously? --- I can see my error info:
ERROR worker.py:844 -- Exception raised in creation task: The actor died because of an error raised in its creation task, ray::Worker.__init__() (pid=xxx, ip=xx...)

RuntimeError: CUDA error: out of memory

CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.

[zhuohan123]

This should be fixed with #397. Please stay tuned!

[hmellor]

Closing as this should now be fixed.