Containerd SystemdCgroup configuration

[aceeric]

Is your feature request related to a problem? Please describe.

This relates to #3106. That issue was closed in favor of this PR for systemd version >= 244.

However, for those running on older systemd versions, this guidance pretty clearly states that not using systemd consistently as the cgroup driver is a potential stability issue.

We can customize the kubelet configuration by specifying kubelet-arg: --cgroup-driver=systemd in the rke2 config file. But to get this configuration into the containerd toml we have to place a complete .toml.tmpl file into the /var/lib/rancher/rke2/agent/etc/containerd/ directory, according to this. This means we have to generate this file ourselves. This file contains selinux interpolation from the rke2 config file and registry mirror configuration from the registries.yaml. It is extremely awkward to have to do all this templating ourselves - to build a correct .toml.tmpl file - just to get SystemdCgroup = true in there.

It seems much more natural to provide the ability for the rke2 config to accept something like cgroup-driver: [cgroupfs | systemd] and then use that to consistently configure the kubelet and containerd.

Describe the solution you'd like
Provide the ability for the rke2 config to accept something like cgroup-driver: [cgroupfs | systemd] and then use that to consistently configure the kubelet and containerd. Or since you already support the kubelet config, maybe containerd-systemd-cgroup-enabled or something like that.

Describe alternatives you've considered

1. Create a one-off containerd toml.tmpl with SystemdCgroup = true, and selinux configured redundantly with what is in the rke2 config and the registry mirrors configured redundantly with what is in the registries.yaml. Build this file using Ansible jinja templating using the template embedded in the k3s go code.
2. Have multiple .toml.tmpl files based on selinux and registry mirrors.

[brandond]

What distro and release are you on, and what version of systemd does it have? Older versions didn't support cpuset cgroup delegation, and were not suitable for use as the cgroup driver.

Ref: https://github.com/opencontainers/runc/blob/main/docs/cgroup-v2.md#systemd

[aceeric]

# cat /etc/*release
CentOS Linux release 7.9.2009 (Core)

# systemctl --version
systemd 219
+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN

[brandond]

Yes, that's too old and should not be used as per the runc documentation I linked above.

[aceeric]

I get you - but - we don't always have control over these things. Are you saying that using systemd as the cgroup driver on this version of systemd for kubelet and containerd running under the RKE2 system service doesn't work? Or doesn't have the intended effect? (I'm out of my depth wrt cpuset delegation - I was taking the containerd guidance as gold source...)

[brandond]

Upstream docs also encourage you to move to cgroup v2 but I'm pretty sure el7 doesn't support that, does it? You're on a very old OS, a lot of the guidance you'll find out there won't be relevant until you go to a newer distro release with modern cgroups and systemd versions.

[aceeric]

Thanks