Should `/var/lib/rancher/k3s/agent/etc/containerd/config-v3.toml.d` be documented?

[adelton]

I recently needed to amend the K3s' containerd configuration, specifically I wanted to add

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc]
  cgroup_writable = true

to enable support for running systemd-based containers with hostUsers: false, for freeipa/freeipa-container#725.

I followed https://docs.k3s.io/advanced#configuring-containerd and https://docs.k3s.io/advanced#base-template and created /var/lib/rancher/k3s/agent/etc/containerd/config-v3.toml.tmpl with content

{{ template "base" . }}
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc]
  cgroup_writable = true

I then saw the containerd fail to start with /var/lib/rancher/k3s/agent/containerd/containerd.log having a stream of

containerd: failed to unmarshal TOML: toml: table runc already exists
containerd: failed to unmarshal TOML: toml: table runc already exists
containerd: failed to unmarshal TOML: toml: table runc already exists

messages.

I checked /var/lib/rancher/k3s/agent/etc/containerd/config.toml and it indeed already has section

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"

there, so there were two [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc]s there in the resulting config.toml. Looking at https://github.com/k3s-io/k3s/blob/main/pkg/agent/templates/templates.go, there is currently no way to customize that default section.

For a while it seemed I would have to copy the whole template from https://github.com/k3s-io/k3s/blob/main/pkg/agent/templates/templates.go (instead of using the recommended {{ template "base" . }} approach) but then I noticed

imports = ["/var/lib/rancher/k3s/agent/etc/containerd/config-v3.toml.d/*.toml"]

at the top of /var/lib/rancher/k3s/agent/etc/containerd/config.toml.

Being brave I did not even try to read what it does, I just created /var/lib/rancher/k3s/agent/etc/containerd/config-v3.toml.d/cgroups.toml with

version = 3
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc]
  cgroup_writable = true

... and it seems to do the trick, containerd starts and systemd-based containers with hostUsers: false also work fine.

I consider drop-in files to be easier than fiddling with templates, so I wonder if the approach which seems to work for me is supported and will work in the future as well, and if yes, whether it should be documented next to (potentially above of) the current https://docs.k3s.io/advanced#base-template section.

[dereknola]

This is a new feature that got added in April releases and has not been documented yet. See k3s-io/k3s#13680.

[brandond]

As Derek said, this was a recent community contribution and hasn't hit the docs yet. PR welcome!

[adelton]

Ah, thanks for the pointer.

@fidencio, have you got any preference for how the support for the imports should be documented?

[fidencio]

@fidencio, have you got any preference for how the support for the imports should be documented?

@adelton, long time no see, and what a pleasure to see you here!
No preference. If you happen to open a PR, just tag me there , please.

[adelton]

Filed #567 as a first draft.