Support for CUDA MPS

[ozancaglayan]

What I'd like:
Nvidia device plugin and the GPU Operator supports a third form of GPU workload sharing called CUDA MPS. Right now, it does not seem that bottlerocket supports configuring of the device plugin for MPS workloads.

Any alternatives you've considered:

• Time slicing
• MIG

Thanks

[yeazelm]

Hello @ozancaglayan, Thanks for cutting this issue! I'd like to know a bit more about how MPS would work better than Time Slicing or MIG for your use case. There isn't a lot of existing documentation for using MPS in k8s so any details on how you would like to use MPS over Time Slicing or MIG would be great to help us understand what might be useful to implement.

It might be possible to install the MPS server in a container and use it for scheduling workloads, but this would skip the scheduling decisions that k8s normally would do.

[ozancaglayan]

Hi there,

GPU operator already supports running the MPS daemon but it is discouraged by bottlerocket for simplicity reasons. Thats why I now switched to configuring kubelet and NVDP manually through bottlerocket guidance for time slicing and it works. But I got a horrible latency with time-slicing.

SO, It is mainly to test latency & throughput trade-off to compare time-slicing and MPS. There are evidences that tell that MPS is better than time-slicing

• https://www.reddit.com/r/kubernetes/comments/1190utm/mps_better_than_mig_and_timeslicing_for/
• https://stackoverflow.com/questions/78653544/why-use-mps-time-slicing-or-mig-if-nvidias-defaults-have-better-performance

so we'd like to evaluate it for real for one of our workloads.

[ozancaglayan]

Hi,

We've tested this with out workload and got remarkably well latencies compared to time-slicing. Trying to install support for this through NVDP plugin on Bottlerocket but got no luck so far.

44384 daemon.go:97] "Staring MPS daemon" resource="nvidia.com/gpu"                                                                                                         │
│ 8:01.432791   44384 daemon.go:156] "SELinux enabled, setting context" path="/mps/nvidia.com/gpu/pipe" context="system_u:object_r:container_file_t:s0"                                    │
│ 8:01.433063   44384 main.go:212] Failed to start MPS daemon: error setting SELinux context: lsetxattr(label=system_u:object_r:container_file_t:s0) /mps/nvidia.com/gpu/pipe: permission  │
│ 8:01.433079   44384 main.go:131] Failed to start one or more MPS deamons. Retrying in 30s...                                                                                             │
│ 8:31.460601   44384 main.go:227] Failed to remove .ready file: remove /mps/.ready: no such file or directory                                                                             │
│ 8:31.460679   44384 main.go:229] Stopping MPS daemons.                                                                                                                                   │
│ 8:31.461091   44384 main.go:83] error stopping plugins from previous run: error sending quit message: failed to start NVIDIA MPS command: exec: "nvidia-cuda-mps-control":

[yeazelm]

I've got an implementation working across my forks for adding MPS support. I'll start cleaning up the code to be ready for review. This should end up being a few new settings to enable MPS:

• settings.kubelet-device-plugins.nvidia.device-sharing-strategy=mps which enables MPS, defaults to 2 replicas
• settings.kubelet-device-plugins.nvidia.mps.replicas which can change how many replicas are created
• settings.kubelet-device-plugins.nvidia.mps.rename_by_default which is similar to the time slicing setting

I need to work out a few more minor issues due to the still experimental support for MPS in the k8s device plugin but I have gotten MPS working. Some of these manifest as something we need to solve for Bottlerocket like the coordination between the device plugin and the MPS control daemon. But the limitations around only supporting full GPUs and combining other things like MIG will still apply no matter what we try to do. So this experimental support has some edge cases that won't go away, even if we solve the coordination problem.

This change involves a lot of pieces so I'm going to document them here to help me track what happens next:

• Kernel kit change to add the MPS binaries Add mps control server and control binaries to NVIDIA kmod packages bottlerocket-kernel-kit#347
• A core kit change that starts building mps-control-daemon in the device plugin and sets up a service to start/stop it based upon settings to configure it. There is a template change as well for the k8s device plugin to handle setting up MPS configuration.
• A bottlerocket-settings-sdk change to add those settings described above.
• A change to take the new bottlerocket-settings-sdk once we have it released to add the settings to NVIDIA variants.

I might have missed a few things but I'll get the changes worked out soon and will update this issue as things come up.

Coordinating the device plugin and MPS control daemon

One thing that was tricky to iron out was the coordination to disable the MPS control daemon when MPS isn't configured. I have a scrapy way to do this by having a systemd dropin that just "daemonizes" sleep infinity when MPS isn't configured, and then switches to the proper binary when MPS is configured. This saves some memory overhead by not having the MPS daemon always running when not configured. There might be a better way to do this with enhancing our systemd control logic simliar to how host-containers works. The problem stems from try-restart not doing anything for a loaded/inactive service. But start/stop will fail if Conditions fail for a unit, so our current tools to control a systemd service that should only run with some settings isn't generic enough for MPS. I think it would be a nice improvement to make the host-containers logic more generic but that may not be strictly required for this. The other concern is that NVIDIA calls this decoupled logic out as a gap in their support, so I want to leave it open to adapt to any improvements they make with coordinating these daemons as well.