Cgroup v2 support

[tchoumbe-91]

Hello,

I'm having troubles running rke2 v1.22.7 version on a cluster based on RHEL 8 nodes with cgroup v2.

I followed the indications from k8s website :
https://kubernetes.io/docs/setup/production-environment/container-runtimes/#cgroup-v2

So, I use :

• a config.toml.tmpl file to use Systemd as a cgroup-driver for Containerd
• a config.yaml file to use Systemd as a cgroup-driver for Kubelet

And I set grub with these parameters :

• systemd.legacy_systemd_cgroup_controller=0
• systemd.unified_cgroup_hierarchy=1

The Server node is running without troubles, but the Agent node has kubelet stucked in a starting loop with the following error :
level=error msg="Kubelet exited: exit status 1"

Do you have any idea about this issue ?

[brandond]

You shouldn't need to change anything in RKE2 to support cgroupv2. The correct cgroup driver should be detected and configured automatically.

level=error msg="Kubelet exited: exit status 1"

There should be other errors prior to this; can you attach the actual log contents?

[brandond]

We test things with their default, shipping configurations. RKE2 agent works fine on EL8 as long as you use the default cgroup driver. That is the configuration that we have tested and validated.

[gfrankliu]

I am running k3s on a single node Debian 11. By default, Debian 11 has cgroups v2 enabled, but k3s log still says:

Apr 16 22:29:29 rls-d k3s[2496026]: W0416 22:29:29.717352 2496026 manager.go:159] Cannot detect current cgroup on cgroup v2

and Kubelet on the same node uses cgroupfs driver by default:

Apr 16 22:29:29 rls-d k3s[2496026]: time="2022-04-16T22:29:29Z" level=info msg="Running kubelet --address=0.0.0.0 --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=cgroupfs  ...

Is there a bug in the cgroup version auto detection code?

[brandond]

No. cgroupfs is a valid driver for cgroup v2, and as discussed above, the only one that k3s or rke2 will use unless you manually configure it to do otherwise.

[gfrankliu]

I just tried installing k3s on a single node Debian 11:

curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644 --disable traefik --kubelet-arg cgroup-driver=systemd

First, I see pods has "RESTARTs" count increasing. After a few minutes, all pods are in CrashLoopBackOff state.

$ kubectl get pod -A
NAMESPACE     NAME                                      READY   STATUS    RESTARTS      AGE
kube-system   coredns-96cc4f57d-rgk92                   1/1     Running   0             2m53s
kube-system   local-path-provisioner-84bb864455-jql9b   1/1     Running   2 (28s ago)   2m53s
kube-system   metrics-server-ff9dbcb6c-zfhq9            1/1     Running   3 (23s ago)   2m53s
$ kubectl get pod -A
NAMESPACE     NAME                                      READY   STATUS             RESTARTS      AGE
kube-system   coredns-96cc4f57d-rgk92                   1/1     Running            0             7m5s
kube-system   local-path-provisioner-84bb864455-jql9b   0/1     CrashLoopBackOff   3 (38s ago)   7m5s
kube-system   metrics-server-ff9dbcb6c-zfhq9            0/1     CrashLoopBackOff   5 (35s ago)   7m5s

[brandond]

Not sure how that's related to the cgroup driver. Please take this back to the k3s repo and post more information on your problem, including any logs or kubectl describe output that might show why your pods are crashing.

[tchoumbe-91]

Hi,

I finally managed to run rke2-agent properly on my RHEL 8 node using cgroup v2 through systemd.

The kernel documentation indicates the following restriction on the CPU controller :

WARNING: cgroup2 doesn’t yet support control of realtime processes and the cpu controller can only be enabled when all RT processes are in the root cgroup. Be aware that system management software may already have placed RT processes into nonroot cgroups during the system boot process, and these processes may need to be moved to the root cgroup before the cpu controller can be enabled.

Indeed, the rtkit-daemon was running on the node, so the related cgroup slice made this RT process not in the root cgroup.
This is probably the main reason behind the kubelet logs about the cgroup while kubelet kept crashing.

I added the following element to the rke2 service configuration to move the MainPID service into the root cgroup :
ExecStartPre=/bin/sh -c "systemctl show --property MainPID rtkit-daemon | cut -d= -f2 > /sys/fs/cgroup/cgroup.procs"

Then, the rke2-agent service starts working as expected.

Note that this solution suppose that rtkit-daemon:

• is the only RT process on the node,
• has not started any other RT process on the node yet.