DevContainer cannot start with "HostRequirements.gpu = optional" when GPU driver installed but GPU doesn't exist.

[jonnyyu]

Thanks for working on optional GPU support. The configuration is exactly what I'm looking for!, but the current GPU check logic still not work in my use case.

According to #173, Dev container checks if docker info contains nvidia runtimes for GPU driver support.

I use Linux EC2 instance with CUDA driver installed, I switch between non-GPU or GPU instance types depending on the type of current work (coding or training). So on non-GPU instance, the nvidia container runtime still available since it is installed.

A suggestion is, Nvidia container runtime also provides a cli tool nvidia-container-cli, can be used to get real GPU info.

on GPU instance:

➜  ~ nvidia-container-cli info
NVRM version:   510.47.03
CUDA version:   11.6

Device Index:   0
Device Minor:   0
Model:          Tesla T4
Brand:          Nvidia
GPU UUID:       GPU-630ef986-c301-d90b-3581-8afa4becebc3
Bus Location:   00000000:00:1e.0
Architecture:   7.5

on non-GPU instance:

➜  ~ nvidia-container-cli info
nvidia-container-cli: initialization error: nvml error: driver not loaded

(exit code 1)

https://github.com/NVIDIA/nvidia-container-runtime

Thanks!

[chrmarti]

One weakness of the current approach is that it is specific to the Nvidia driver. Would using docker run -it --rm --gpus all alpine:latest echo OK work for testing for the GPU?

[jonnyyu]

@chrmarti Yes, the command will success on GPU and fail on nonGPU instance.

On non-GPU instance:

docker run -it --rm --gpus all alpine:latest echo OK
docker: Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: Running hook #0:: error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: initialization error: nvml error: driver not loaded: unknown.

on GPU instance:

docker run -it --rm --gpus all alpine:latest echo OK
OK