process_group/gloo: support CUDA tensors

[d4l3k]

This makes torchft's ProcessGroupGloo correctly support CUDA tensors so we can use Gloo with torchtitan for allreduces.

I also found a bug in Gloo, apparently torchft's test suite is more in depth than the one in core haha pytorch/pytorch#152645

Test plan:

torchtitan on remote cluster

pytest torchft/process_group_test.py -k gloo_apis

[H-Huang]

cool! I also wonder if it would be possible to use torchtitan for cpu-only training. That could be cool to show a demo on a single host

[fegin]

@H-Huang ye, I'm also wondering if we can do CPU training. But I'm not sure if FSDP can work with CPU memory.

[d4l3k]

I've only tested this with HSDP using NCCL for most of titan and only Gloo+CUDA for the allreduce

Gloo does CPU blocking so I expect that we'll have some significant performance hit since this won't overlap compute+networking. I have some thoughts on how to work around that but tbd