fix(manifests): add NVML init container for NVIDIA GPU Operator support

[bitflicker64]

Summary

On clusters with the NVIDIA GPU Operator, Kepler was mounting the whole driver-container root FS from /run/nvidia/driver. That path includes the driver's glibc, so dynamic linking breaks when NVML loads.

This PR switches to the usual pattern: an init container copies only libnvidia-ml.so* into an emptyDir, and Kepler mounts that at /usr/local/nvidia/lib64 with LD_LIBRARY_PATH pointed there. NVML libs only, no driver glibc.

Closes #2484

What changed

• manifests/helm/kepler/values.yaml: chart-only knobs under daemonset.nvidia.* (driverPath, nvmlInitImage). Not under config.experimental.gpu.*, because the configmap dumps the whole config tree into /etc/kepler/config.yaml and the binary only knows enabled / idlePower / dcgmEndpoint there.
• manifests/helm/kepler/templates/daemonset.yaml: when config.experimental.gpu.enabled is true, add the nvidia-libs init container, emptyDir mount, LD_LIBRARY_PATH, hostPath + emptyDir volumes (sizeLimit: 200Mi).
• manifests/k8s/daemonset.yaml: same pattern, always on (raw manifest has no values system).
• .pre-commit-config.yaml: local helm-lint-gpu hook for the gpu.enabled=true variant. Default state stays on the existing helmlint hook. No extra CI job.
• docs/user/installation.md: how to enable GPU power monitoring with the GPU Operator, including overrides and the flag coupling note.

Design notes

• config.experimental.gpu.enabled is intentionally coupled to the chart plumbing. Flip that flag and you get both the binary path and the init container. There is no separate daemonset.nvidia.enabled.
• NVIDIA_VISIBLE_DEVICES / NVIDIA_MIG_MONITOR_DEVICES stay unconditional. They were always set, and the runtimeClassName: nvidia workaround depends on them.
• LD_LIBRARY_PATH is a static value. K8s only expands env vars defined earlier in the same pod spec, not the image env, so we cannot prepend whatever the image already had. The dynamic linker still searches /lib and /usr/lib by default.
• Init uses set -e and does not mask cp with || true. A full emptyDir or a permission error should fail the pod at init, not start Kepler with a half-copied library.
• The copy pass globs the known driver lib dirs (usr/lib64, usr/lib/x86_64-linux-gnu, usr/lib/aarch64-linux-gnu) instead of running find over the driver root. The driver-container root FS has its own /proc mounted inside it, so a full walk is slow and exits nonzero on unreadable proc entries, which set -e (correctly) treats as fatal. An unknown future layout degrades to the same "No NVML libraries found" path as a non-GPU node instead of failing init.
• cp -P (preserve symlinks), not cp -L. Real NVIDIA drivers are a symlink chain (libnvidia-ml.so -> .so.1 -> .so.580.x). Leftover driver versions on the host make cp -L blow past the 200Mi sizeLimit; cp -P keeps the chain cheap.
• Init runs as UID/GID 0 with capabilities dropped, so root-owned driver files are readable without depending on the busybox image's USER.

Review feedback folded in

• Chart knobs moved out of config.* into daemonset.nvidia.* (Copilot).
• Explicit runAsUser / runAsGroup: 0 (Copilot). Kept readOnlyRootFilesystem: true (mount points are created by the runtime before the remount; emptyDir writes do not need a writable rootfs).
• cp -P instead of cp -L (shellyco-code), with host numbers below.
• Dropped the dedicated helm-lint CI job; use the local pre-commit hook instead (vprashar).
• Trimmed heavy init-script comments in the chart; long rationale lives here and in the commit message (vprashar).
• User docs + flag-coupling callout (vprashar).
• Init no longer walks the driver root FS with find. On a real GPU Operator cluster it traversed the driver container's /proc for minutes and crash-looped on unreadable entries (laurall974's H100 report). The copy pass now globs the known lib dirs only.

Testing

helm lint and helm template for both gpu.enabled states. Disabled render has no init container / nvml emptyDir path. Enabled ConfigMap only has enabled / idlePower / dcgmEndpoint under experimental.gpu.

On a real cluster with a real GPU

Single-node k3s v1.36.2 (Ubuntu 24.04), NVIDIA RTX 3050, driver 580.173.02. Chart installed from this branch with --set config.experimental.gpu.enabled=true, kepler image 5495d5d-20260725174242. Manifest unmodified.

Non-GPU node path (no /run/nvidia/driver on the host):

Check	Result
Init container	Completed, exitCode 0, logs "No NVML libraries found at /run-nvidia"
hostPath DirectoryOrCreate	created /run/nvidia/driver, node still schedules
Init securityContext as applied by kubelet	readOnlyRootFilesystem: true, drop: [ALL], runAsUser/Group: 0 — container started and exited 0
kepler	NVML init failed: ERROR_LIBRARY_NOT_FOUND, no GPUs discovered (this is the log from #2484)
Pod	Running, ready, 0 restarts, 0 GPU metrics

GPU node path — /run/nvidia/driver/usr/lib64 staged as the GPU Operator exposes the driver container root: real NVML symlink chain plus the driver's own userspace (libc.so.6, ld-linux-x86-64.so.2, libcuda.so.580.173.02, libnvidia-glcore, libnvidia-opencl), 218 MB total.

Check	Result
Init container	Copied 3 of 3 NVML library file(s), exitCode 0
emptyDir at /usr/local/nvidia/lib64	2.2 MB, symlink chain intact, NVML only — no glibc, no libcuda
NVML resolved by kepler (/proc/<pid>/maps)	/usr/local/nvidia/lib64/libnvidia-ml.so.580.173.02
kepler	discovered GPU name="NVIDIA GeForce RTX 3050 Laptop GPU", NVML initialized device_count=1
Exported	kepler_node_gpu_watts 3.224, kepler_node_gpu_joules_total 3148.016, kepler_node_gpu_info 1
Host nvidia-smi at the same moment	3.22 W

Second GPU, strict isolation

Tesla T4, driver 580.95.05, system NVML moved out of the loader's search path so the copied libraries were the only possible source.

Check	Result
Init script from helm template, byte-for-byte under busybox:1.36.1	exit 0, Copied 3 of 3
cp -P vs cp -L on the same tree	2.3 MB vs 6.9 MB
kepler with the emptyDir	NVML initialized, kepler_node_gpu_watts 13.709
Control, LD_LIBRARY_PATH unset	ERROR_LIBRARY_NOT_FOUND, 0 GPU metrics

Earlier host-only validation (driver 580.159.03) agreed on the sizing: leftover multi-version host was ~152 MiB with cp -P vs ~307 MiB with cp -L.

GPU Operator /proc regression, multi-node

4-node kubeadm v1.30.14 (3x Ubuntu 24.04, 1x Fedora 43), containerd. Driver root staged on one worker the way the operator exposes it, including a live procfs mounted at /run/nvidia/driver/proc, the mount the H100 report tripped on. Chart from this branch, --set config.experimental.gpu.enabled=true, no nodeSelector, so it schedules on every node including the tainted control plane.

Check	Result
Previous revision (find), driver node	init exit 1 with find: /run-nvidia/proc/1/task/1/fdinfo: Permission denied (same lines as the H100 log), Init:CrashLoopBackOff
This revision (glob), driver node	exit 0 in under a second, Copied 3 NVML library file(s)
emptyDir	symlink chain intact (.so -> .so.1 -> .so.580.65.06)
kepler on driver node	Running 1/1, 2170 metrics, live RAPL zones
3 nodes without /run/nvidia (incl. Fedora and control plane)	No NVML libraries found, exit 0, kepler Running
RHEL-style layout (usr/lib64 instead of x86_64-linux-gnu)	Copied 3, kepler Running

T4 re-check with the new script taken byte-for-byte from the manifest: real driver libs (580.95.05) staged in the operator layout, script exit 0, then NVML loaded strictly from the copied directory: nvmlInit SUCCESS, device discovered, live power readable.

Not covered

• A real GPU Operator install. The driver root was staged by hand from the host driver rather than the operator's driver container, and device nodes reached the pod via privileged: true rather than runtimeClassName: nvidia. PSA is untested.

[Copilot]

Pull request overview

This PR updates Kepler’s Kubernetes manifests (raw + Helm) to support NVIDIA GPU Operator deployments by avoiding a direct mount of the driver container root filesystem into the Kepler container. Instead, it uses an init container to copy only libnvidia-ml.so* into an emptyDir mounted at /usr/local/nvidia/lib64, and adjusts LD_LIBRARY_PATH accordingly.

Changes:

• Add an nvidia-libs init container that copies NVML libraries from the host driver path into an emptyDir.
• Update the DaemonSet to mount the copied NVML libraries into the main Kepler container and point LD_LIBRARY_PATH at that location.
• Add a CI job to lint and render the Helm chart with config.experimental.gpu.enabled=true.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
manifests/k8s/daemonset.yaml	Raw DaemonSet: add init container + emptyDir NVML library mounting and update LD_LIBRARY_PATH.
manifests/helm/kepler/values.yaml	Helm values: add knobs for driver path and init image used by the NVML-copy pattern.
manifests/helm/kepler/templates/daemonset.yaml	Helm DaemonSet template: gate init container, volumes, mount, and LD_LIBRARY_PATH on gpu.enabled.
.github/workflows/pr-checks.yaml	Add Helm lint/template CI coverage for the gpu.enabled=true variant.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

[bitflicker64]

rebased after the 3 new commits today

[vprashar2929]

@bitflicker64 Can you squash into a single commit?

[bitflicker64]

@bitflicker64 Can you squash into a single commit?

done!

[shellyco-code]

Hi @bitflicker64! Thanks for this PR, this is a great approach for handling the NVML glibc conflicts.

I was testing the init container logic and noticed one quick thing that might cause issues: Because find matches both the actual library and its symlinks (e.g., .so, .so.1), using cp -L will dereference all of them and copy the full 50MB file multiple times into the emptyDir. This will consume ~150MB+ immediately, which risks hitting your 200Mi sizeLimit and evicting the pod (especially if the host has leftover files from a driver update).

Could you swap that to cp -P or cp -d (or just cp -a) to preserve the symlinks instead of duplicating the file contents? Thanks again for the contribution!

[bitflicker64]

Could you swap that to cp -P or cp -d (or just cp -a) to preserve the symlinks instead of duplicating the file contents? Thanks again for the contribution!

Makes sense

[vprashar2929]

@bitflicker64 A few notes:

1. The new helm-lint CI feels unnecessary to me. I would focus on how we can keep things consistent and rely on single mechanism for validation i.e pre-commit hooks. Prefer looking to a small local pre-commit hook if pre-commit doesn't provide us something out of box.
2. The init-container script commenst are accurate but much heavy for the chart. Worth trimming to short "why" and leaving the deep rationale in the PR body. Easier to maintain when we revisit this.
3. Chart pattern looks fine. We gate pod plumbing on the app config flag rather than a separate daemonset.nvidia.enabled I think that's the right UX here. Just call that out so nobody expects to enable the binary flag without init path
4. Would be good to mention briefly in user docs about the changes this PR introduces

[vprashar2929]

@bitflicker64 By any chance have we tested this with actual GPU node?

[bitflicker64]

Hi @bitflicker64! Thanks for this PR, this is a great approach for handling the NVML glibc conflicts.

I was testing the init container logic and noticed one quick thing that might cause issues: Because find matches both the actual library and its symlinks (e.g., .so, .so.1), using cp -L will dereference all of them and copy the full 50MB file multiple times into the emptyDir. This will consume ~150MB+ immediately, which risks hitting your 200Mi sizeLimit and evicting the pod (especially if the host has leftover files from a driver update).

Could you swap that to cp -P or cp -d (or just cp -a) to preserve the symlinks instead of duplicating the file contents? Thanks again for the contribution!

Pushed with cp -P. Verified on real nvidia driver 580.159.03 symlink chain
(libnvidia-ml.so → .so.1 → .so.580.159.03). On a host with the current
driver + 3 leftover versions, cp -P = 153 MB (fits in 200Mi cap), cp -L
= 307 MB (~50% over → pod eviction). cp -P carries leftover symlinks
through as symlinks; cp -L dereferences them into 50MB copies.

[bitflicker64]

@vprashar2929 latest push (e9612e1) covers your notes:

helm-lint CI is gone. GPU-enabled lint is a local pre-commit hook (helm-lint-gpu); default state still uses the existing helmlint hook.

Init-container comments are trimmed to short "why" in the chart. Longer rationale is in the PR body / commit message.

Flag coupling is called out in values.yaml and the new install docs: flipping config.experimental.gpu.enabled turns on the daemonset.nvidia.* init plumbing too. No separate enable flag.

User docs: "Enabling GPU Power Monitoring (NVIDIA GPU Operator)" in docs/user/installation.md.

GPU validation on a real host (driver 580.159.03, RTX 3050). Not a full GPU Operator cluster:

• ran the PR's init script in busybox:1.36.1 against a simulated driver dir
• leftover multi-version host: cp -P ~152 MiB (under 200 MiB), cp -L ~307 MiB (over the cap)
• clean production chain: 3 files, ~2.2 MiB, symlinks preserved
• loaded the isolated libs with ctypes and with LD_LIBRARY_PATH / pynvml: nvmlInit ok, 1 GPU, live power ~3.25 W

Still missing: a real Kepler pod on a GPU Operator node (scheduling / runtimeClass / PSA). If there's a preferred lab path, I can re-run there.

[nikimanoledaki]

@laurall974 might be able to test this 👀

[bitflicker64]

@laurall974 might be able to test this 👀

sounds good to me !

[bitflicker64]

@vprashar2929 @nikimanoledaki re: testing on an actual GPU node — deployed this on single-node k3s v1.36.2 with a real RTX 3050 (driver 580.173.02). Chart unmodified, --set config.experimental.gpu.enabled=true, kepler image 5495d5d-20260725174242.

No driver on the node: init container exits 0 with "No NVML libraries found at /run-nvidia", DirectoryOrCreate creates the path, kepler logs NVML init failed: ERROR_LIBRARY_NOT_FOUND / no GPUs discovered, pod stays Running ready with 0 restarts. That WARN is the exact log from #2484, so it doubles as a repro. Also relevant to @copilot's note: the kubelet applied the init securityContext as written (readOnlyRootFilesystem: true, drop: [ALL], runAsUser/Group: 0) and the container started fine.

Driver staged GPU-Operator-style (real NVML chain plus the driver's own libc.so.6, ld-linux, libcuda, libnvidia-glcore, libnvidia-opencl — 218 MB):

init: "Copied 3 of 3 NVML library file(s)", exit 0
emptyDir at /usr/local/nvidia/lib64: 2.2M, symlink chain intact, NVML only

/proc/<pid>/maps:  /usr/local/nvidia/lib64/libnvidia-ml.so.580.173.02
kepler: "discovered GPU name=NVIDIA GeForce RTX 3050" / "NVML initialized device_count=1"

kepler_node_gpu_watts  3.224
host nvidia-smi        3.22 W

218 MB in, 2.2 MB out, no glibc. NVML resolved from the emptyDir and nowhere else.

Also ran it on a Tesla T4 (driver 580.95.05) with the system NVML moved out of the loader's search path: same result, and a control with LD_LIBRARY_PATH unset gives ERROR_LIBRARY_NOT_FOUND with zero GPU metrics. The init script from helm template ran byte-for-byte under busybox:1.36.1 there too — cp -P 2.3 MB vs cp -L 6.9 MB on the same tree (@shellyco-code's point on real driver libs).

Not covered: a real GPU Operator install. I staged the driver root by hand and the device nodes reached the pod via privileged: true, not runtimeClassName: nvidia. Mixed-cluster scheduling, PSA and multi-node are untested — @laurall974 that's the remaining gap.

Rebased onto latest main, squashed to a single commit.

[vprashar2929]

@laurall974 Can we have this patch tested?

[laurall974]

@bitflicker64 @vprashar2929

Set Up

So I was able to test it on real GPU-Operator-style setup: three H100 nodes running the containerized NVIDIA driver via nvidia-driver-daemonset.
We deployed the PR chart unmodified except for scheduling/security overrides and config.experimental.gpu.enabled=true. Values:

daemonset:
  nodeSelector:
    nvidia.com/gpu.present: "true"
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: kubernetes.io/hostname
                operator: In
                values: [ <gpu-node-1>, <gpu-node-2>, <gpu-node-3> ]
  tolerations:
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule
    - key: node.kubernetes.io/unschedulable
      operator: Exists
      effect: NoSchedule
  securityContext:
    privileged: true
    runAsNonRoot: false
    runAsUser: 0
    runAsGroup: 0
config:
  experimental:
    gpu:
      enabled: true

Result

The nvidia-libs init container does not complete. It runs for ~5 minutes, then terminates with exit code 1 (reason: Error), and the pod goes into CrashLoopBackOff (BackOff restarting failed container nvidia-libs), so kepler stays in PodInitializing and never starts.

Init container status:

"terminated": { "exitCode": 1, "reason": "Error",
  "startedAt": "2026-08-07T13:17:10Z", "finishedAt": "2026-08-07T13:22:16Z" }
Restart Count: 3

The only init container output is repeated permission-denied lines from find traversing the driver container's /proc:

kubectl logs my-kepler-2p9h2 --container nvidia-libs 

find: /run-nvidia/proc/1/task/1/fdinfo: Permission denied
find: /run-nvidia/proc/1/fdinfo: Permission denied
find: /run-nvidia/proc/2/task/2/fdinfo: Permission denied

Conclusion

No Copied N of N or No NVML libraries found line was reached. The main container's /usr/local/nvidia/lib64 emptyDir was never populated and I could not exec into kepler (still initializing).

[bitflicker64]

@laurall974, makes sense. I’ll look into it. Thanks for testing it!!

[bitflicker64]

@laurall974 thanks for the detailed report !! reproduced it, fixed it, pushed (also rebased on main).

root cause: /run/nvidia/driver is the driver container's full root FS, and that includes its own mounted /proc. the init script's find walked that procfs (that's your ~5 minutes) and hit the unreadable fdinfo entries. the count pass hid those errors with 2>/dev/null but the copy pass didn't, so find returned nonzero and set -e turned that into exit 1 -> CrashLoopBackOff. exactly the lines in your log.

fix: dropped find entirely. the script now globs the known driver lib dirs (usr/lib64, usr/lib/x86_64-linux-gnu, usr/lib/aarch64-linux-gnu) and never touches the rest of the driver root. if some future driver image uses a different layout, it falls into the existing "No NVML libraries found" path instead of failing init.

repro + verification: 4-node kubeadm v1.30.14, driver root staged with a live procfs mounted at /run/nvidia/driver/proc:

check	find (old)	glob (new)
init container, driver node	exit 1, find: /run-nvidia/proc/1/task/1/fdinfo: Permission denied, CrashLoopBackOff	exit 0 in under a second, Copied 3 NVML library file(s)
symlink chain in emptyDir	-	intact (.so -> .so.1 -> .so.580.65.06)
3 nodes without /run/nvidia	-	No NVML libraries found, exit 0, kepler Running
rhel-style usr/lib64 layout	-	Copied 3, kepler Running

also re-ran the T4 strict-isolation check with the new script taken byte-for-byte from the manifest: real driver 580.95.05 staged in the operator layout, script exit 0, then NVML initialized strictly from the copied dir (nvmlInit SUCCESS, device found, live power read). full details in the updated PR description.

when you get a chance, could you rerun it on the H100 setup? thanks again for catching this!!

[laurall974]

@bitflicker64

Setup (rerun on H100 setup after fix)

GPU-Operator-style setup: H100 nodes running the containerized NVIDIA driver via nvidia-driver-daemonset. The PR chart was deployed unmodified except for scheduling/security overrides and config.experimental.gpu.enabled=true. Same values as the previous run:

daemonset:
  nodeSelector:
    nvidia.com/gpu.present: "true"
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: kubernetes.io/hostname
                operator: In
                values: [ <gpu-node-1>, <gpu-node-2>, <gpu-node-3> ]
  tolerations:
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule
    - key: node.kubernetes.io/unschedulable
      operator: Exists
      effect: NoSchedule
  securityContext:
    privileged: true
    runAsNonRoot: false
    runAsUser: 0
    runAsGroup: 0
config:
  experimental:
    gpu:
      enabled: true

I pulled the rebased branch, redeployed, and observed the pods over time on two separate sets of nodes.

Result: init container is fixed

The nvidia-libs init container now completes successfully — the original ~5-minute hang and init-time CrashLoopBackOff are gone. On every pod I checked:

$ kubectl logs my-kepler-vrkgx --container nvidia-libs
Copied 3 NVML library file(s) to /nvml-libs

$ kubectl logs my-kepler-fd7ql --container nvidia-libs
Copied 3 NVML library file(s) to /nvml-libs

kepler proceeds past init and starts.

But: the main kepler container now crash-loops on the liveness probe

First set of nodes. Shortly after deploy the pods were Running but already accumulating restarts:

$ kubect get pods | rg my-kepler
my-kepler-2lfpg   1/1   Running   248 (6m20s ago)   23h
my-kepler-49njv   1/1   Running   4                 23h
my-kepler-vrkgx   1/1   Running   1 (23h ago)       23h

my-kepler-2lfpg had 248 restarts at 23h. The container log ends in a graceful shutdown, not an error:

time=2026-08-10T14:00:27.212Z level=INFO source=prometheus/collector/power_collector.go:267 msg="Collected unified power data" collector=power duration=1.929835895s
time=2026-08-10T14:00:27.427Z level=INFO source=internal/service/run.go:56 msg="shutting down" service=monitor
time=2026-08-10T14:00:27.427Z level=INFO source=internal/monitor/monitor.go:182 msg="shutting down monitor" service=monitor
time=2026-08-10T14:00:27.427Z level=INFO source=cmd/kepler/main.go:70 msg="Graceful shutdown completed"

Second set of nodes. Suspecting a node-specific issue, I redeployed on three other nodes. Init was again fine:

$ kubectl get pods | grep kepler
my-kepler-8qlk6   1/1   Running   7 (56s ago)     24m
my-kepler-fd7ql   1/1   Running   7 (2m54s ago)   24m
my-kepler-gk277   1/1   Running   2 (11m ago)     24m

$ kubectl logs my-kepler-fd7ql --container nvidia-libs
Copied 3 NVML library file(s) to /nvml-libs

Left running for ~18 hours, restart counts climbed and one pod fell into CrashLoopBackOff:

$ kubectl get pods
NAME              READY   STATUS             RESTARTS          AGE
my-kepler-8qlk6   1/1     Running            206 (54s ago)     18h
my-kepler-fd7ql   0/1     CrashLoopBackOff   202 (4m52s ago)   18h
my-kepler-gk277   1/1     Running            113 (4m49s ago)   18h

Pod events and container state (my-kepler-fd7ql)

The main container's last terminated state is a clean exit, and the events show the kills come from the liveness probe:

State:          Waiting → CrashLoopBackOff
Last State:     Terminated  Reason: Completed  Exit Code: 0
Restart Count:  7
Liveness:       http-get http://:http/metrics delay=10s timeout=1s period=60s #success=1 #failure=3

Warning  Unhealthy  Liveness probe failed: Get "http://10.100.36.79:28282/metrics":
                     context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Normal   Killing    Container kepler failed liveness probe, will be restarted
Warning  BackOff    Back-off restarting failed container kepler

my-kepler-8qlk6 shows the same pattern at higher volume — 206 restarts, 637 Unhealthy events over 18h, all context deadline exceeded on /metrics.

Additional observations

• Increasing the liveness probe has timeout might fix it (I did not test it) but maybe it is something worth investigating more.