Sysbox installation on Rancher Kubernetes (RKE2) fails

[FiRo-IT]

Hi Guys,

I discovered the same problem as in issue 380

It's a RKE2 Cluster with Kubernetes Version 1.31.7 (but also tried with Version 1.31.1 and 1.28.15)
OS are Ubuntu 20.04

The cluster and rancher are installed as described in this document.

After enter kubectl label nodes node2 sysbox-install=yes and kubectl apply -f https://raw.githubusercontent.com/nestybox/sysbox/master/sysbox-k8s-manifests/sysbox-install.yaml the sysbox-deploy-k8s-d956r pod are in a crashloopbackoff with the error:

Detected Kubernetes version v1.28
Adding K8s taint "sysbox-runtime=not-running:NoSchedule" to node ...
node/node2 modified
Deploying Kubelet config agent on the host ...
Running Kubelet config agent on the host (will restart Kubelet and temporary bring down all pods on this node for ~1 min) ...
Job for kubelet-config-helper.service failed because the control process exited with error code. See "systemctl status kubelet-config-helper.service" and "journalctl -xe" for details.

root@node2:~# systemctl status kubelet-config-helper:

● kubelet-config-helper.service - Kubelet config service
Loaded: loaded (/lib/systemd/system/kubelet-config-helper.service; static; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2025-04-09 12:22:25 UTC; 3min 1s ago
Process: 43182 ExecStart=/bin/sh -c /usr/local/bin/kubelet-config-helper.sh (code=exited, status=5)
Main PID: 43182 (code=exited, status=5)

Apr 09 12:22:25 node2 sh[43183]: SUB
Apr 09 12:22:25 node2 sh[43183]: 1 ...
Apr 09 12:22:25 node2 sh[43183]: + systemctl stop LOAD ACTIVE SUB 1
Apr 09 12:22:25 node2 sh[43296]: Failed to stop LOAD.service: Unit LOAD.service not loaded.
Apr 09 12:22:25 node2 sh[43296]: Failed to stop ACTIVE.service: Unit ACTIVE.service not loaded.
Apr 09 12:22:25 node2 sh[43296]: Failed to stop SUB.service: Unit SUB.service not loaded.
Apr 09 12:22:25 node2 sh[43296]: Failed to stop 1.service: Unit 1.service not loaded.
Apr 09 12:22:25 node2 systemd[1]: kubelet-config-helper.service: Main process exited, code=exited, status=5/NOTINSTALLED
Apr 09 12:22:25 node2 systemd[1]: kubelet-config-helper.service: Failed with result 'exit-code'.
Apr 09 12:22:25 node2 systemd[1]: Failed to start Kubelet config service.

journalctl -xe:

root@node2:~# journalctl -xe

Subject: Unit process exited
Defined-By: systemd
Support: http://www.ubuntu.com/support

An ExecStart= process belonging to unit kubelet-config-helper.service has exited.

The process' exit code is 'exited' and its exit status is 5.
Apr 09 12:11:21 node2 systemd[1]: kubelet-config-helper.service: Failed with result 'exit-code'.
Subject: Unit failed
Defined-By: systemd
Support: http://www.ubuntu.com/support

The unit kubelet-config-helper.service has entered the 'failed' state with result 'exit-code'.
Apr 09 12:11:21 node2 systemd[1]: Failed to start Kubelet config service.
Subject: A start job for unit kubelet-config-helper.service has failed
Defined-By: systemd
Support: http://www.ubuntu.com/support

A start job for unit kubelet-config-helper.service has finished with a failure.

The job identifier is 5182 and the job result is failed.
Apr 09 12:11:21 node2 systemd[1]: cri-containerd-b969eba95345185a22cfe13920873e26d3b1a665db4d1c02a9d3047522852381.scope: Succeeded.
Subject: Unit succeeded
Defined-By: systemd
Support: http://www.ubuntu.com/support

The unit cri-containerd-b969eba95345185a22cfe13920873e26d3b1a665db4d1c02a9d3047522852381.scope has successfully entered the 'dead' state.
Apr 09 12:11:21 node2 systemd[1]: run-k3s-containerd-io.containerd.runtime.v2.task-k8s.io-b969eba95345185a22cfe13920873e26d3b1a665db4d1c02a9d3047522852381-rootfs.mount: Succeeded.
Subject: Unit succeeded
Defined-By: systemd
Support: http://www.ubuntu.com/support

The unit run-k3s-containerd-io.containerd.runtime.v2.task-k8s.io-b969eba95345185a22cfe13920873e26d3b1a665db4d1c02a9d3047522852381-rootfs.mount has successfully entered the 'dead' state.
Apr 09 12:11:24 node2 snapd[36332]: daemon.go:548: gracefully waiting for running hooks
Apr 09 12:11:24 node2 snapd[36332]: daemon.go:550: done waiting for running hooks
Apr 09 12:11:27 node2 snapd[36332]: overlord.go:518: Released state lock file
Apr 09 12:11:27 node2 snapd[36332]: daemon stop requested to wait for socket activation
Apr 09 12:11:27 node2 systemd[1]: snapd.service: Succeeded.
Subject: Unit succeeded
Defined-By: systemd
Support: http://www.ubuntu.com/support

The unit snapd.service has successfully entered the 'dead' state.`

I've tried to figure out if I need special requirements for RKE2 but as described in https://github.com/nestybox/sysbox/blob/master/docs/user-guide/install-k8s.md#kubernetes-worker-node-requirements it should work out of the box.

Am I missing something?

This is a test-cluster. So if anybody needs access to it let me know.

Thanks in advance and best regards
Franz

[FiRo-IT]

I switched to microk8s and tried the sysbox installation again -> same error

after edit /var/snap/microk8s/current/args/kubelet to use cri-o with this lines:

container-runtime-endpoint=unix:///var/run/crio/crio.sock
runtime-request-timeout=10m
cgroup-driver="systemd"

the installation of sysbox are able to complete and now im also able to use containers with systemd

[ctalledo]

Hi @FiRo-IT, thanks for reporting the issue.

We used to support installation of Sysbox on RKE2 but it's been a while since we've tested on it, so it's not totally surprising the installation program no longer works (it's a tricky program because it must "drop" Sysbox and CRI-O into the RKE cluster, then reconfigure K8s to use CRI-O instead of containerd, and restart the Kubelet, and then wait for all pods to come back up).

Unfortunately we have no cycles currently to support RKE2, so I'll likely go ahead and remove it from the docs.

Having said this, the program that installs Sysbox on a K8s cluster is a bash script (see here). The code that is specific to installing on RKE2 is here.

You can play with it and tweak it, we are happy to take any contributions that would make it work on RKE or microk8s.

[mrpastewart]

Having said this, the program that installs Sysbox on a K8s cluster is a bash script (see here). The code that is specific to installing on RKE2 is here.

You can play with it and tweak it, we are happy to take any contributions that would make it work on RKE or microk8s.

I will take a look at the bash script in regards to microk8s and see what we can do. I am currently setting up sysbox across a docker swarm cluster (which is done) and now i am doing a microk8s cluster and will get it working if i can, will post back.