[Bug]: NVIDIA tasks fail on headless hosts without /dev/nvidia-modeset

[peterschmidt85]

Steps to reproduce

Run an NVIDIA task on a headless host where CUDA works but /dev/nvidia-modeset is not present.

Example CloudRift host where this reproduced:

region: ap-east-tw-kn-1
instance type: rtx49-10c-kn.1
GPU: NVIDIA GeForce RTX 4090
driver: 575.57.08

The host has the usual NVIDIA compute devices:

ls /dev/nvidia0 /dev/nvidiactl /dev/nvidia-uvm /dev/nvidia-uvm-tools
ls /dev/nvidia-caps

It does not have /dev/nvidia-modeset:

ls /dev/nvidia-modeset

Plain Docker CUDA works:

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi -L

The same host fails when Docker is asked for the NVIDIA capabilities currently requested by dstack-shim:

docker run --rm \
  --gpus 'all,"capabilities=gpu,utility,compute,graphics,video,display,compat32"' \
  nvidia/cuda:12.4.1-base-ubuntu22.04 \
  nvidia-smi -L

Actual behaviour

The container fails before the user command starts:

nvidia-container-cli: mount error: stat failed: /dev/nvidia-modeset: no such file or directory

This prevents normal CUDA workloads from running, even though the host can run nvidia-smi and CUDA containers when display is not requested.

Expected behaviour

CUDA/PyTorch workloads should run on NVIDIA compute hosts that do not expose display devices.

display should not be required for ordinary compute workloads.

dstack version

master

Server logs

nvidia-container-cli: mount error: stat failed: /dev/nvidia-modeset: no such file or directory

Additional information

dstack-shim currently requests this Docker container.DeviceRequest.Capabilities set for NVIDIA GPUs:

gpu,utility,compute,graphics,video,display,compat32

NVIDIA documents display as X11 display support. The documented default driver capabilities are utility,compute: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/docker-specialized.html

A safe fix is to keep the current capability set on the first attempt, then retry without only display if NVIDIA Container Runtime fails specifically because /dev/nvidia-modeset is missing. This should not apply to non-NVIDIA vendors or explicit GPUDevices handling.

[hiSandog]

This looks like a good regression-test candidate because the failure is narrowed to “NVIDIA tasks fail on headless hosts without /dev/nvidia-modeset”. If a minimal fixture can exercise /dev/nvidia-modeset and assert the expected behavior around dstack-shim, it would make the eventual fix much easier to keep from regressing.