cgroups v2 - systemd is not starting in the container

[kevin-leong]

Hello,

I am having issues with cgroups v2 with Ubuntu 21.10 (which recently went to cgroups v2 by default) and running K3s v1.21.5+k3s2. The freeipa container, which uses systemd, does not start when using cgroups v2 is being used, but cgroups v1 is working fine.

Can someone please help take a look at this issue and let me know how this can be fixed so cgroups v2 can be used?

To switch to using cgroups v1, I use the kernel argument "systemd.unified_cgroup_hierarchy=0".

Also to note, I clone this repo locally and build my own aarch64 version of the container using "Dockerfile.centos-8-stream" directly since there no aarch64 version on dockerhub for example.

Thanks.

[K3S Version]

root@kube1:/# kubectl get nodes
NAME                    STATUS   ROLES                       AGE   VERSION
kube1   				Ready    control-plane,etcd,master   30h   v1.21.5+k3s2
kube2   				Ready    control-plane,etcd,master   30h   v1.21.5+k3s2
kube3   				Ready    control-plane,etcd,master   30h   v1.21.5+k3s2
kube4   				Ready    <none>                      30h   v1.21.5+k3s2

[snippet of my statefulset config]

      containers:
      - name: ipa1
        image: XXX:5000/freeipa
        env:
        - name: IPA_SERVER_HOSTNAME
          value: XXX
        - name: PASSWORD
          value: XXX
        args:
        - --domain=XXX
        - --realm=XXX
        - --idstart=100000
        - --no-ntp
        - --no-ssh
        - --no-sshd
        - --unattended
        - --setup-kra
        ports:
        - name: dns
          containerPort: 53
        - name: http
          containerPort: 80
        - name: https
          containerPort: 443
        - name: ldap
          containerPort: 389
        - name: ldaps
          containerPort: 636
        - name: kerberos
          containerPort: 88
        - name: kpasswd
          containerPort: 464
        volumeMounts:
        - name: data
          mountPath: /data
        - name: cgroups
          mountPath: /sys/fs/cgroup
          readOnly: true
        - name: run
          mountPath: /run
        - name: tmp
          mountPath: /tmp
      volumes:
      - name: cgroups
        hostPath:
          path: /sys/fs/cgroup
      - name: run
        emptyDir:
          medium: Memory
      - name: tmp
        emptyDir:
          medium: Memory

[cgroups v2]

root@kube1:/# cat /proc/cmdline 
coherent_pool=1M 8250.nr_uarts=1 snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=1 bcm2708_fb.fbwidth=0 bcm2708_fb.fbheight=0 bcm2708_fb.fbswap=1 smsc95xx.macaddr=DC:A6:32:BF:6B:40 vc_mem.mem_base=0x3eb00000 vc_mem.mem_size=0x3ff00000  dwc_otg.lpm_enable=0 console=ttyS0,115200 console=tty1 root=LABEL=writable rootfstype=ext4 elevator=deadline rootwait fixrtc quiet splash

root@kube1:/# ls /sys/fs/cgroup
cgroup.controllers	cgroup.threads	       dev-mqueue.mount  memory.stat			system.slice
cgroup.max.depth	cpu.pressure	       init.scope	 misc.capacity			user.slice
cgroup.max.descendants	cpuset.cpus.effective  io.pressure	 sys-fs-fuse-connections.mount
cgroup.procs		cpuset.mems.effective  io.stat		 sys-kernel-config.mount
cgroup.stat		cpu.stat	       kubepods		 sys-kernel-debug.mount
cgroup.subtree_control	dev-hugepages.mount    memory.pressure	 sys-kernel-tracing.mount

root@kube1:/# kubectl logs pod/ipa1-0
root@kube1:/# kubectl exec -it ipa1-0 -- /bin/bash
[root@ipa1-0 /]# ps -aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.1  0.0  20360  7384 ?        Ss   02:11   0:00 /usr/sbin/init --show-status=false
root          47  0.0  0.0   5012  1200 ?        S    02:11   0:00 /usr/bin/coreutils --coreutils-prog-shebang=t
root          48  2.2  0.0  20288  3468 pts/0    Ss   02:15   0:00 /bin/bash
root          62  0.0  0.0  25948  3612 pts/0    R+   02:15   0:00 ps -aux
[root@ipa1-0 log]# systemctl status
Failed to connect to bus: No such file or directory
[root@ipa1-0 /]# ls /sys/fs/cgroup
cgroup.controllers	cgroup.threads	       dev-mqueue.mount  memory.stat			system.slice
cgroup.max.depth	cpu.pressure	       init.scope	 misc.capacity			user.slice
cgroup.max.descendants	cpu.stat	       io.pressure	 sys-fs-fuse-connections.mount
cgroup.procs		cpuset.cpus.effective  io.stat		 sys-kernel-config.mount
cgroup.stat		cpuset.mems.effective  kubepods		 sys-kernel-debug.mount
cgroup.subtree_control	dev-hugepages.mount    memory.pressure	 sys-kernel-tracing.mount

[cgroups v1]

root@kube1:/# cat /proc/cmdline 
coherent_pool=1M 8250.nr_uarts=1 snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=1 bcm2708_fb.fbwidth=0 bcm2708_fb.fbheight=0 bcm2708_fb.fbswap=1 smsc95xx.macaddr=DC:A6:32:BF:6B:40 vc_mem.mem_base=0x3eb00000 vc_mem.mem_size=0x3ff00000  dwc_otg.lpm_enable=0 console=ttyS0,115200 console=tty1 root=LABEL=writable rootfstype=ext4 elevator=deadline rootwait fixrtc quiet splash systemd.unified_cgroup_hierarchy=0

root@kube1:/# ls /sys/fs/cgroup
blkio  cpu  cpuacct  cpu,cpuacct  cpuset  devices  freezer  hugetlb  memory  misc  net_cls  net_cls,net_prio  net_prio	perf_event  pids  rdma	systemd  unified

root@kube1:/# kubectl logs pod/ipa1-0
....
Mon Oct 18 17:32:06 UTC 2021 /usr/sbin/ipa-server-configure-first update-self-ip-address
FreeIPA server does not run DNS server, skipping update-self-ip-address.
FreeIPA server started.

root@kube1:/# kubectl exec -it ipa1-0 -- /bin/bash
[root@ipa1-0 /]# ps -aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.3  0.1  22048  9192 ?        Ss   02:26   0:01 /usr/sbin/init --show-status=false
root          47  0.0  0.0   5012  1200 ?        S    02:26   0:00 /usr/bin/coreutils --coreutils-prog-shebang=tail /usr/bin/tail --silent -n 0 -f --retry /var/log/ipa-s
root          62  0.1  0.1  28612 10452 ?        Ss   02:26   0:00 /usr/lib/systemd/systemd-journald
root          66  0.1  0.1  66600 12984 ?        Ss   02:26   0:00 /usr/sbin/sssd -i --logger=files
dbus          67  0.0  0.0  30704  4640 ?        Ss   02:26   0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-on
root          70  0.0  0.0  10564  2276 ?        Ss   02:26   0:00 /usr/sbin/oddjobd -n -p /run/oddjobd.pid -t 300
root          79  0.0  0.0 134592  4528 ?        Ssl  02:26   0:00 /usr/sbin/gssproxy -D
root          88  0.4  0.1  64792 13268 ?        S    02:26   0:01 /usr/libexec/sssd/sssd_be --domain implicit_files --uid 0 --gid 0 --logger=files
...........
[root@ipa1-0 /]# systemctl status ipa
● ipa.service - Identity, Policy, Audit
   Loaded: loaded (/usr/lib/systemd/system/ipa.service; enabled; vendor preset: disabled)
   Active: active (exited) since Tue 2021-10-19 02:32:06 JST; 2min 19s ago
  Process: 81 ExecStart=/usr/sbin/ipactl start (code=exited, status=0/SUCCESS)
 Main PID: 81 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 49296)
   Memory: 0B
   CGroup: /system.slice/k3s.service/system.slice/ipa.service

Oct 19 02:26:33 ipa1-0 systemd[1]: Starting Identity, Policy, Audit...
Oct 19 02:32:06 ipa1-0 ipactl[81]: ipa: INFO: The ipactl command was successful
Oct 19 02:32:06 ipa1-0 ipactl[81]: Starting Directory Service
Oct 19 02:32:06 ipa1-0 ipactl[81]: Starting krb5kdc Service
Oct 19 02:32:06 ipa1-0 ipactl[81]: Starting kadmin Service
Oct 19 02:32:06 ipa1-0 ipactl[81]: Starting httpd Service
Oct 19 02:32:06 ipa1-0 ipactl[81]: Starting ipa-custodia Service
Oct 19 02:32:06 ipa1-0 ipactl[81]: Starting pki-tomcatd Service
Oct 19 02:32:06 ipa1-0 ipactl[81]: Starting ipa-otpd Service
Oct 19 02:32:06 ipa1-0 systemd[1]: Started Identity, Policy, Audit.
[root@ipa1-0 /]# ls /sys/fs/cgroup
blkio  cpu  cpu,cpuacct  cpuacct  cpuset  devices  freezer  hugetlb  memory  misc  net_cls  net_cls,net_prio  net_prio	perf_event  pids  rdma	systemd  unified

[kevin-leong]

Hello - appreciate if someone can take a look at this and provide some guidance. Thanks.

[adelton]

This seems similar to failures under docker that @danryu reported in #429, also on Ubuntu 21.10. You might want to join forces to figure out how systemd is supposed to be run in containers on those hosts.

[adelton]

Per https://github.com/actions/virtual-environments GitHub Actions currently don't provide 21.10, so I am unable to debug this easily.

[adelton]

I assume #429 is reasonable place to have further discussion.