Multi-GPU Support

[jw1912]

No description provided.

[Disservin]

this ready to be tested?

[jw1912]

@Disservin it works correctly (verified by Viren on 1,2,3,4x 4090s) but atm it requires a large batch size to be a good speedup because the gradient accumulation code is basic and suboptimal.

[Disservin]

so the interconnect is limiting too much for it to be worth much ?
on my server the pytorch multigpu is no speedup but for vondele it is since it has nvlink between the gpus

[jw1912]

@Disservin yes, but if you increase the batch size by several times the transfers become infrequent enough that it is a speedup on normal multigpu (transfers are just weight gradients/values so no dependence on batch size ofc).
It would be cool to get a test on a system with nvlink

[jw1912]

I will be trying some alternative strategies to reduce the latency of the interconnect.