Difference in containerd config between workers and masters

[rlex]

Environmental Info:
K3s Version:

k3s version v1.24.2+k3s1 (b0ed1348)
go version go1.18.1

Node(s) CPU architecture, OS, and Version:

Linux node-1 5.15.0-39-generic #42-Ubuntu SMP Thu Jun 9 23:42:32 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:
3 masters, embedded etcd, 5 workers

Describe the bug:
While debugging gvisor not launching on control-plane nodes, i noticed strange behaviour of k3s-generated containerd config. I add some gvisor-related records (here is template, just in case). So template gets picked up at launch and generates configs. Now for interesting part.
Here is master config:

#Template needs to be re-synced with each containerd update
#Current version:
#https://github.com/k3s-io/k3s/blob/v1.24.2%2Bk3s1/pkg/agent/templates/templates_linux.go

[plugins.opt]
  path = "/var/lib/rancher/k3s/agent/containerd"
[plugins.cri]
  stream_server_address = "127.0.0.1"
  stream_server_port = "10010"
  enable_selinux = false
  enable_unprivileged_ports = true
  enable_unprivileged_icmp = true
  sandbox_image = "rancher/mirrored-pause:3.6"
[plugins.cri.containerd]
  snapshotter = "overlayfs"
  disable_snapshot_annotations = true
[plugins.cri.containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"
[plugins.cri.containerd.runtimes.runc.options]
	SystemdCgroup = false
[plugins.cri.containerd.runtimes.runsc]
  runtime_type = "io.containerd.runsc.v1"
[plugins.cri.containerd.runtimes.runsc.options]
  SystemdCgroup = false
  TypeUrl = "io.containerd.runsc.v1.options"
  ConfigPath = "/var/lib/rancher/k3s/agent/etc/containerd/runsc.toml"

And here is worker:

#Template needs to be re-synced with each containerd update
#Current version:
#https://github.com/k3s-io/k3s/blob/v1.24.2%2Bk3s1/pkg/agent/templates/templates_linux.go

[plugins.opt]
  path = "/var/lib/rancher/k3s/agent/containerd"
[plugins.cri]
  stream_server_address = "127.0.0.1"
  stream_server_port = "10010"
  enable_selinux = false
  enable_unprivileged_ports = true
  enable_unprivileged_icmp = true
  sandbox_image = "rancher/mirrored-pause:3.6"
[plugins.cri.containerd]
  snapshotter = "overlayfs"
  disable_snapshot_annotations = true
[plugins.cri.containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"
[plugins.cri.containerd.runtimes.runc.options]
	SystemdCgroup = true
[plugins.cri.containerd.runtimes.runsc]
  runtime_type = "io.containerd.runsc.v1"
[plugins.cri.containerd.runtimes.runsc.options]
  SystemdCgroup = true
  TypeUrl = "io.containerd.runsc.v1.options"
  ConfigPath = "/var/lib/rancher/k3s/agent/etc/containerd/runsc.toml"

OSes are same, kernels are same, all rolled from same ansible role, without major differences. There is some differences in k3s config.yaml, but master have some stuff disabled and so on. Same k3s versions, obviously.

So why SystemdCgroup is different, true on workers and false on masters?

Steps To Reproduce:

• Installed K3s
• Copied template so k3s will produce readable config
• Noticed differences

Expected behavior:
Configs should be same?

Actual behavior:
Configs are not same

Additional context / logs:

Backporting

• Needs backporting to older releases

[brandond]

Duplicate of

• systemd cgroup driver detection does not work on servers #5850

[rlex]

Oh huh, hasn't found it! Thanks, will keep track on original one, then