Skip to content

[Bug]: Kubernetes: all node GPUs allocated when none requested #4010

Description

@un-def

Steps to reproduce

Prerequisites:

  • GPU nodes are not tainted (dstack doesn't add tolerations if no GPU requested).
  • An image with NVIDIA_VISIBLE_DEVICES=all baked into, e.g., NVIDIA images or dstackai/base (which is used when image is not set in the run configuration).
type: dev-environment
name: dev-environment
resources:
  gpu: 0
  1. dstack apply
  2. ssh dev-environment
  3. ls -1 /dev/nvidia[0-9]
  4. nvidia-smi -L

Actual behaviour

All GPUs are passed inside the container via the NVIDIA_VISIBLE_DEVICES env var path.

# ls -1 /dev/nvidia[0-9]
/dev/nvidia0
/dev/nvidia1
/dev/nvidia2
/dev/nvidia3
/dev/nvidia4
/dev/nvidia5
/dev/nvidia6
/dev/nvidia7

# nvidia-smi -L
GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-fe56c782-f8db-cc17-fa82-4286037f988b)
GPU 1: NVIDIA H100 80GB HBM3 (UUID: GPU-cba9dcf6-8c94-fa2b-1bf0-f1476941a458)
GPU 2: NVIDIA H100 80GB HBM3 (UUID: GPU-33ac7880-d9d2-c4b7-8ad1-a1c55d5124d6)
GPU 3: NVIDIA H100 80GB HBM3 (UUID: GPU-74a529a1-2939-79e1-7f54-aec2d35756c6)
GPU 4: NVIDIA H100 80GB HBM3 (UUID: GPU-cbf70ec6-5ef2-eadc-510c-ed98ee0fc6ed)
GPU 5: NVIDIA H100 80GB HBM3 (UUID: GPU-37d22e64-5838-4837-a577-916774e2edf7)
GPU 6: NVIDIA H100 80GB HBM3 (UUID: GPU-43d11db3-eb4d-5e26-0b3a-81ed92f37a86)
GPU 7: NVIDIA H100 80GB HBM3 (UUID: GPU-56aec030-3f6e-09b3-10ce-0c9834e72182)

Expected behaviour

No GPUs inside the container.

dstack version

0.20.26

Server logs

Additional information

No response

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions