GPU support for gVisor actors

[eliranw]

Goal

Let actors use NVIDIA GPUs passthrough from their workers.

How to achieve this

We need to inject the relevant info to the actor's OCI spec, and then runsc proxies the driver.

Suggested Approach

1. Install GPU-Operator on the K8s cluster.
2. Request GPU resources in a workerpool. This should mount the ateom with relevant NVIDIA binaries and tooling and expose the GPUs to the workers.
3. When starting, ateom runs nvidia-ctk cdi generate,and takes the device nodes, mounts, hooks, and env and emits them to a CDI file. Hooks run using nvidia-cdi-hook as part of createContainer.
4. Merge into the actor's OCI spec.
5. Ateom runs runsc create to build the sandbox with the relevant GPU devices.

The core tension

Running NVIDIA CDI inside an unprivileged gVisor worker has the following issues:

1. CDI createContainer hooks run in runsc's gofer, which drops to a minimal cap set.
2. update-ldcache hook mounts /proc to run the hook. we need /proc to be unmasked to support this.
3. The current ateom-gvisor worker image is distroless, but the NVIDIA toolkit binaries are glibc-dynamic so they can't be exec'd directly; they can be run via a staged glibc loader from a shared volume.

Net: GPU on the unprivileged worker requires hostUsers: false + procMount: Unmasked and that can break cgorup delegation.

Known costs / open issues

• Toolkit host-path portability. gpu-operator/EKS install at /usr/local/nvidia/toolkit; GKE keeps it under /home/kubernetes/bin/nvidia. Made configurable via env.
• Snapshot/restore with GPU out of scope currently (suspend/resume of a GPU actor).