Custom path for runtimes autodetect

[Ivan-Rosa]

Is your feature request related to a problem? Please describe.
I'm using rke2 with Flatcar OS, using a custom path to install Nvidia toolkit.
Because of this custom path, runtimes autodetect is not finding nvidia runtime.

Describe the solution you'd like
It would be nice to be allowed to add some custom path in

k3s/pkg/agent/containerd/config_linux.go

Line 27 in 85e02e1

runtimesPath = "/usr/local/nvidia/toolkit:/opt/kwasm/bin:/usr/sbin:/usr/local/sbin:/usr/bin:/usr/local/bin"

k3s/pkg/agent/containerd/config_linux.go

Line 61 in 85e02e1

os.Setenv("PATH", runtimesPath)

[brandond]

Where does Flatcar want to put things? Can you symlink them into one of the expected paths?

[vitorsavian]

@brandond maybe could we do something like this to support more paths? since we still will get the paths that we want and the user can add his path to the search too

runtimesPath = "/usr/local/nvidia/toolkit:/opt/kwasm/bin"

originalPath := os.Getenv("PATH")
os.Setenv("PATH", runtimesPath + os.PathListSeparator + originalPath)
extraRuntimes := findContainerRuntimes()
os.Setenv("PATH", originalPath)

[ShylajaDevadiga]

Testing reference: https://docs.k3s.io/advanced#nvidia-container-runtime-support

[mdrahman-suse]

@vitorsavian thanks for sharing the steps here: #10953 (comment). But I am not seeing the expected behavior when I followed the steps. Please advise if I am missing anything...

Steps:

export PATH=/home/ubuntu:$PATH
touch nvidia-container-runtime && chmod +x nvidia-container-runtime
curl -sfL https://get.k3s.io | INSTALL_K3S_COMMIT=e9c529530ca87484845db76a57ef80466135b2ef sh -
kubectl get nodes,pods -A -o wide
sudo cat /var/lib/rancher/k3s/agent/etc/containerd/config.toml | grep nvidia

$ k3s -v
k3s version v1.31.1+k3s-e9c52953 (e9c52953)
go version go1.22.6

ubuntu@ltsInstance2:~$ pwd
/home/ubuntu
ubuntu@ltsInstance2:~$ ls
nvidia-container-runtime

$ sudo cat /var/lib/rancher/k3s/agent/etc/containerd/config.toml
# File generated by k3s. DO NOT EDIT. Use config.toml.tmpl instead.
version = 2

[plugins."io.containerd.internal.v1.opt"]
  path = "/var/lib/rancher/k3s/agent/containerd"
[plugins."io.containerd.grpc.v1.cri"]
  stream_server_address = "127.0.0.1"
  stream_server_port = "10010"
  enable_selinux = false
  enable_unprivileged_ports = true
  enable_unprivileged_icmp = true
  sandbox_image = "rancher/mirrored-pause:3.6"

[plugins."io.containerd.grpc.v1.cri".containerd]
  snapshotter = "overlayfs"
  disable_snapshot_annotations = true



[plugins."io.containerd.grpc.v1.cri".cni]
  bin_dir = "/var/lib/rancher/k3s/data/cni"
  conf_dir = "/var/lib/rancher/k3s/agent/etc/cni/net.d"


[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  SystemdCgroup = true

[plugins."io.containerd.grpc.v1.cri".registry]
  config_path = "/var/lib/rancher/k3s/agent/etc/containerd/certs.d"

@ShylajaDevadiga FYI, When I follow the below steps along with the steps that @vitorsavian has provided, I only see the default path in config.toml (in both previous and latest versions) where nvidia-container-runtime is installed...

• https://docs.k3s.io/advanced#nvidia-container-runtime-support

$ sudo cat /var/lib/rancher/k3s/agent/etc/containerd/config.toml | grep nvidia
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."nvidia"]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."nvidia".options]
  BinaryName = "/usr/bin/nvidia-container-runtime"

[vitorsavian]

@mdrahman-suse I added a PR that will make it easier and it will resolve when running with systemd

but the workaround is to

echo "PATH=${PATH}" | sudo tee -a /etc/systemd/system/k3s.service.env

after exporting the new PATH and then restart k3s withsystemctl restart k3s

[mdrahman-suse]

Validated on master branch with commit e9c5295

Environment

Single server, Ubuntu 22.04

Test steps:

1. Install nvidia-toolkit and nvidia-container-runtime from: https://docs.k3s.io/advanced#nvidia-container-runtime-support
2. Create a dummy executable: touch nvidia-container-runtime && chmod +x nvidia-container-runtime
3. Update PATH: export PATH=/home/ubuntu:$PATH
4. Install k3s

[⚠️ NOTE: The next step is needed when k3s is running with systemd]
5. Add PATH on k3s.service.env: echo "PATH=${PATH}" | sudo tee -a /etc/systemd/system/k3s.service.env
6. Restart k3s.service: sudo systemctl restart k3s
7. Ensure updated path shows in .../containerd/config.toml

Replication:

$ k3s -v
k3s version v1.31.1+k3s1 (452dbbc1)
go version go1.22.6

• Observed default path after restart

$ sudo cat /var/lib/rancher/k3s/agent/etc/containerd/config.toml | grep nvidia
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."nvidia"]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."nvidia".options]
  BinaryName = "/usr/bin/nvidia-container-runtime"

Validation:

$ k3s -v
k3s version v1.31.1+k3s-e9c52953 (e9c52953)
go version go1.22.6

• Observed updated path after restart

$ sudo cat /var/lib/rancher/k3s/agent/etc/containerd/config.toml | grep nvidia
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."nvidia"]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."nvidia".options]
  BinaryName = "/home/ubuntu/nvidia-container-runtime"

[aganesh-suse]

#10998 closing based on this.