[Onprem] Support for Different Type of GPUs + Small Bugfix

[michaelzhiluo]

Adds support for the 1080, 2080, A5000, A6000 GPU types (GPUs NOT supported by Ray's backend) and allows scheduling for such types.

Also, fixes the bug introduced in #1287.

Test:

sky status

Local clusters:
NAME                USER    HEAD_IP        RESOURCES                 COMMAND                         
shelby              ubuntu  3.144.21.13    [{'V100': 1}]             sky exec shelby --gpus V1...    
thomas              ubuntu  3.94.208.15    [{'1080': 1, '2080': 1}]  sky launch -c thomas --gpus... 

• sky launch -c thomas --gpus 1080:1 -- 'echo hi' works and occupies the right placement groups
• sky launch -c shelby --gpus V100:1 -- 'echo hi' still works and occupies the right placement group (+ Ray's automatic GPU placement group)

[infwinston]

Sorry could you explain why is supported_ray_accs needed?
For example, we can sky launch gpunode --gpus M60 --cloud aws and Ray will correctly recognize M60 as a gpu even it's not on the list right?

ubuntu@ip-172-31-82-9:~$ ray status
======== Autoscaler status: 2022-11-01 21:22:40.810182 ========
Node status
---------------------------------------------------------------
Healthy:
 1 ray.head.default
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/4.0 CPU
 0.0/1.0 GPU
 0.0/1.0 M60
 0.0/1.0 accelerator_type:M60
 0.00/17.552 GiB memory
 0.00/8.776 GiB object_store_memory

Demands:
 (no resource demands)

[michaelzhiluo]

Thanks Weilin for the quick review!

When M60 or other GPUs are detected by Ray, it will naturally add a GPU to the placement group under the hood, as you see in your M60 logs.

I tested this with a 1080 GPU yesterday, and the GPU field did not show up. So, to be safe, some of this code handles the cases where the GPU field does not show up.

[infwinston]

Do we understand why 1080 can't not be detected by Ray? I was asking because the list here isn't the cause as M60 is not in it but can be correctly identified.

I traced Ray's code a bit. Looks like how they identify #num_gpu is by the function _autodetect_num_gpus below
https://github.com/ray-project/ray/blob/f39d323ed5916b67042407231f5b91851e8fa0b5/python/ray/_private/resource_spec.py#L268
It's by either GPUtil.getGPUs() which relies on nvidia-smi or /proc/driver/nvidia/gpus/ which relies on nvidia-driver I believe.

Then as long as nvidia-smi or its driver works, 1080 GPU should be correctly detected. Could you share more on why Ray failed?

[michaelzhiluo]

Great find! This is my family member's GPU and its fair to say nvdia-smi was not installed and could be why Ray failed to detect.

[WoosukKwon]

@michaelzhiluo
Ray could detect TITAN V GPUs on my machine:

$ ray status
======== Autoscaler status: 2022-11-01 15:06:23.656898 ========
Node status
---------------------------------------------------------------
Healthy:
 1 node_a25d5edb0e115a8ee02542d40ff6868d055789a81d6041717e19d11e
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/48.0 CPU
 0.0/8.0 GPU
 0.0/1.0 accelerator_type:TITAN
 0.00/320.967 GiB memory
 0.00/141.549 GiB object_store_memory

Demands:
 (no resource demands)

I believe it's a problem due to uninstalled (or misinstalled) nvidia-smi, and we can do nothing for that case.

[michaelzhiluo]

Ok, I will work on the fix to detect if the ray status has GPU or not. It seems that it has nothing to do with what types of GPUs it is, but how the Ray detects GPU. Thanks @infwinston @WoosukKwon

[Michaelvll]

Ok, I will work on the fix to detect if the ray status has GPU or not. It seems that it has nothing to do with what types of GPUs it is, but how the Ray detects GPU. Thanks @infwinston @WoosukKwon

For the ray clusters, we can pass --num-gpus to the ray start in the ray yaml to let the ray cluster know how many gpus on each node. Just as what we did for the GCP:

skypilot/sky/templates/gcp-ray.yml.j2

Line 175 in f542327

ray stop; ray start --disable-usage-stats --head --port=6379 --object-manager-port=8076 --autoscaling-config=~/ray_bootstrap_config.yaml {{"--resources='%s'" % custom_resources if custom_resources}} --num-gpus=$SKY_NUM_GPUS || exit 1;

[michaelzhiluo]

Thanks Zhanghao for the great suggestion! Just tested and it works, no need to care about whether Ray detects the gpu or not.

[infwinston]

I left some comments and questions. Hope they help. Others please chime in as I'm not super familiar with on-prem codes and also not sure how to properly test it.

[infwinston]

Do we want to add TITAN Xp (used in a RISE server)?

Also do you think nvidia-smi --list-gpus is more robust for detecting #num-gpus as it's agnostic to gpu names?

weichiang@freddie:~$ nvidia-smi --list-gpus
GPU 0: NVIDIA TITAN Xp (UUID: GPU-67995609-af7b-27e6-2024-0f5f3c837c1a)
GPU 1: NVIDIA TITAN Xp (UUID: GPU-7dc58de3-9a7a-3e5f-6231-c1ac72af9e0d)
GPU 2: NVIDIA TITAN Xp (UUID: GPU-6dcf1597-36aa-b71c-5e4d-713931d79f9b)
GPU 3: NVIDIA TITAN Xp (UUID: GPU-a35c7b48-c7a8-016a-aa46-f82a654bc9a4)
GPU 4: NVIDIA TITAN Xp (UUID: GPU-72bfeabf-998e-ae6e-7ab2-4b0c066869ec)
GPU 5: NVIDIA TITAN Xp (UUID: GPU-6b3b0b2a-89db-24b4-46b5-aab244bcaad8)

weichiang@blaze:~$ nvidia-smi --list-gpus
GPU 0: Tesla P100-PCIE-16GB (UUID: GPU-7942cb79-543a-c8fe-03e8-172ed446f612)
GPU 1: Tesla P100-PCIE-16GB (UUID: GPU-09e76941-9a87-b1b4-8ef7-864c8d98c4af)
GPU 2: Tesla P100-PCIE-16GB (UUID: GPU-a9edaeaa-5c02-2b26-2ad8-7a5d51f0d1ac)
GPU 3: Tesla P100-PCIE-16GB (UUID: GPU-49a19b7d-5a7e-7ee5-8168-c4e77d648998)
GPU 4: Tesla P100-PCIE-16GB (UUID: GPU-b3b998a7-b941-040e-68f6-5f533bcf5636)
GPU 5: Tesla P100-PCIE-16GB (UUID: GPU-8fc7187f-350a-1b62-2ba2-51466153dd06)
GPU 6: Tesla P100-PCIE-16GB (UUID: GPU-7c8c9f76-d06e-2d3d-2f63-873fc9925ecf)
GPU 7: Tesla P100-PCIE-16GB (UUID: GPU-90773d83-1578-d5fb-ccc0-a30b7c0b4e66)

[romilbhardwaj]

+1, we should think about how we can do away with this list and use just nvidia-smi to parse the resources available. If this PR is time-sensitive, we can do it in a different PR

[michaelzhiluo]

We can do this in a separate Pr imo. There has to be a better way to detect GPUs, even with onprem cluster wihtout nvidia-smi installed ( as was this code's assumption, works on all machines with linux/unix). Very interested in discussion for this.

[infwinston]

I think it should be a safe assumption? If nvidia-smi is not installed, very likely Ray will also fail to schedule CUDA jobs.

If we want less assumption, Ray's approach of listing /proc/driver/nvidia/gpus
https://github.com/ray-project/ray/blob/f39d323ed5916b67042407231f5b91851e8fa0b5/python/ray/_private/resource_spec.py#L281 should be better.
It looks pretty robust (similar to lspci, only assume nvidia-driver is installed) and name-agnostic.

weichiang@blaze:~$ ls /proc/driver/nvidia/gpus
0000:04:00.0  0000:06:00.0  0000:07:00.0  0000:08:00.0  0000:0c:00.0  0000:0d:00.0  0000:0e:00.0  0000:0f:00.0
weichiang@blaze:~$ cat /proc/driver/nvidia/gpus/0000\:04\:00.0/information
Model: 		 Tesla P100-PCIE-16GB
IRQ:   		 722
GPU UUID: 	 GPU-7942cb79-543a-c8fe-03e8-172ed446f612
Video BIOS: 	 86.00.41.00.06
Bus Type: 	 PCIe
DMA Size: 	 47 bits
DMA Mask: 	 0x7fffffffffff
Bus Location: 	 0000:04:00.0
Device Minor: 	 0
GPU Excluded:	 No

This will auto cover TITAN Xp and TITAN V that currently not in the list but used in RISE's and Woosuk's server? Does that make sense?

[michaelzhiluo]

It seems a bit different on Windows (how Ray handles it), which is the OS used by one of my family members (along with nvidia-smi not being available despite being able to use the GPU). I'll look deeper into it for a future PR.

[infwinston]

Some follow-up suggestions. If you think this PR is too urgent, feel free to ignore them and raise issues to fix in the future.

[infwinston]

I think it should be a safe assumption? If nvidia-smi is not installed, very likely Ray will also fail to schedule CUDA jobs.

If we want less assumption, Ray's approach of listing /proc/driver/nvidia/gpus
https://github.com/ray-project/ray/blob/f39d323ed5916b67042407231f5b91851e8fa0b5/python/ray/_private/resource_spec.py#L281 should be better.
It looks pretty robust (similar to lspci, only assume nvidia-driver is installed) and name-agnostic.

weichiang@blaze:~$ ls /proc/driver/nvidia/gpus
0000:04:00.0  0000:06:00.0  0000:07:00.0  0000:08:00.0  0000:0c:00.0  0000:0d:00.0  0000:0e:00.0  0000:0f:00.0
weichiang@blaze:~$ cat /proc/driver/nvidia/gpus/0000\:04\:00.0/information
Model: 		 Tesla P100-PCIE-16GB
IRQ:   		 722
GPU UUID: 	 GPU-7942cb79-543a-c8fe-03e8-172ed446f612
Video BIOS: 	 86.00.41.00.06
Bus Type: 	 PCIe
DMA Size: 	 47 bits
DMA Mask: 	 0x7fffffffffff
Bus Location: 	 0000:04:00.0
Device Minor: 	 0
GPU Excluded:	 No

This will auto cover TITAN Xp and TITAN V that currently not in the list but used in RISE's and Woosuk's server? Does that make sense?

[infwinston]

LGTM. Please consider filing an issue for supporting more GPU types :)