Bug in runc causes kubectl exec failure after systemd daemon-reload

[clrxbl]

Is your feature request related to a problem? Please describe.
Yes. The runc version in k3s' containerd version 1.6.6 contains a regression that prevents anyone from executing a command and attaching to the container's TTY (exec -it) whenever someone runs systemctl daemon-reload. Alternatively, the user may run into this issue on SELinux-enforced systems.

containerd/containerd#7219

Describe the solution you'd like
I am not sure how k3s maintainers usually handle these issues, but I would very much like to see a k3s release that updates runc to 1.1.4.

Describe alternatives you've considered
Downgrade k3s to 1.23.

Additional context

❯ kubectl exec -it -n kube-system cilium-6lqp9 -- cilium status
Defaulted container "cilium-agent" out of: cilium-agent, mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), wait-for-node-init (init), clean-cilium-state (init)
error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "b67e6e00172071996430dac5c97352e4d0c9fa3b3888e8daece5197c4649b4d1": OCI runtime exec failed: exec failed: unable to start container process: open /dev/pts/0: operation not permitted: unknown

[brandond]

Is this a problem with the runc binary that we're including, or the runc module used by containerd? They're not necessarily the same.

[brandond]

I'll think it should be easy to update runc in both locations, but I echo this comment: containerd/containerd#7219 (comment)

[brandond]

According to opencontainers/runc#3551 (comment) it has to do with the systemd group driver, and systemd mucking about with permissions when it reloads. Interesting...

[clrxbl]

Is this a problem with the runc binary that we're including, or the runc module used by containerd? They're not necessarily the same.

Unsure.

[kolyshkin]

Is this a problem with the runc binary that we're including, or the runc module used by containerd? They're not necessarily the same.

It's the binary (unless your software is using runc/libcontainer/cgroups/systemd package to create systemd units on a cgroup v1 system, in which case it's also the vendored packages).

[est-suse]

Validated on with /

Environment Details

Infrastructure

• Cloud
• Hosted

Node(s) CPU architecture, OS, and Version:

PRETTY_NAME="Ubuntu Jammy Jellyfish (development branch)"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04 (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Cluster Configuration:

1 server 1 agent

Testing Steps

$ /var/lib/rancher/k3s/data/current# bin/runc -v

$ /var/lib/rancher/k3s/data/current# bin/runc -v