Probe with exec fails because no cgroup directory is found

[rogue73]

Description

I'm user of kubernetes distro k3s an someone else and myself notice a problem that maybe come from runc.

We have a open issue in the k3s project: https://github.com/k3s-io/k3s/issues/13459?utm_source=copilot.com

The problem exist since k3s version 1.34.3.
And one of the developers proposed to open it as an issue of runc because the major change in this area between K3s v1.34.2 and v1.34.3 was the bump of runc from v1.3.3 to v1.4.0.

If you try to deploy docker k8s driver but failed to probe readiness with exec since cgroup sllice not found.
The container alive and I can see the logs of the container. It seems that the cgroupv2 slice path is not resolved correctly?.

Steps to reproduce the issue

$ docker buildx create --bootstrap --name=kube \
    --driver=kubernetes \
    --platform=linux/amd64 \
    --node=builder-amd64 \
    --driver-opt=nodeselector=kubernetes.io/arch=amd64
$ kubectl get pods
> NAME                             READY   STATUS    RESTARTS   AGE
> builder-amd64-7b74b6dfd6-g874s   0/1     Running   0          15m
$ kubectl describe pod builder-amd64-7b74b6dfd6-g874s
> ...
> Warning  Unhealthy  16m                 kubelet            Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = failed to exec in container: failed to start exec "74fb2dee11586bc3c897fef8ce86e38a3b319c86f1d9fbaa812083c2a82c7fd1": OCI runtime exec failed: exec failed: unable to start container process: error starting setns process: can't open cgroup: openat2 /sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0eade01_e660_4eb4_975a_203120127fc3.slice/cri-containerd-bf5fd9d00b6d21e44e0a56383386ee9f1898ebe0cc718a05815e8e93ca7ecb10.scope: no such file or directory

The /sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0eade01_e660_4eb4_975a_203120127fc3.slice/cri-containerd-bf5fd9d00b6d21e44e0a56383386ee9f1898ebe0cc718a05815e8e93ca7ecb10.scope does not exist.

but the container is OK:

$ kubectl logs pods/builder-amd64-7b74b6dfd6-g874s
> time="2026-01-11T08:03:41Z" level=info msg="auto snapshotter: using overlayfs"
> time="2026-01-11T08:03:41Z" level=warning msg="using host network as the default"
> time="2026-01-11T08:03:41Z" level=info msg="found worker \"5bxqduetmhw1niuwxa69x7qpx\", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:builder-amd64-7b74b6dfd6-g874s org.mobyproject.buildkit.worker.network:host org.mobyproject.buildkit.worker.oci.process-mode:sandbox org.mobyproject.buildkit.worker.selinux.enabled:false org.mobyproject.buildkit.worker.snapshotter:overlayfs], platforms=[linux/amd64 linux/amd64/v2 linux/amd64/v3 linux/amd64/v4 linux/386]"
> time="2026-01-11T08:03:41Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
> time="2026-01-11T08:03:41Z" level=info msg="found 1 workers, default=\"5bxqduetmhw1niuwxa69x7qpx\""
> time="2026-01-11T08:03:41Z" level=warning msg="currently, only the default worker can be used."
> time="2026-01-11T08:03:41Z" level=info msg="running server on /run/buildkit/buildkitd.sock"

Describe the results you received and expected

I receive:

Warning Unhealthy 16m kubelet Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = failed to exec in container: failed to start exec "74fb2dee11586bc3c897fef8ce86e38a3b319c86f1d9fbaa812083c2a82c7fd1": OCI runtime exec failed: exec failed: unable to start container process: error starting setns process: can't open cgroup: openat2 /sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0eade01_e660_4eb4_975a_203120127fc3.slice/cri-containerd-bf5fd9d00b6d21e44e0a56383386ee9f1898ebe0cc718a05815e8e93ca7ecb10.scope: no such file or directory

I expect: Container starts without error and probes did not fail.

What version of runc are you using?

1.4.0

Host OS information

# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

# k3s -v
k3s version v1.34.3+k3s1 

Host kernel information

# uname -a
Linux k3stest 6.1.0-42-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.159-1 (2025-12-30) x86_64 GNU/Linux

[kolyshkin]

Alas this is hardly actionable since we don't have a way to reproduce.

[brandond]

Also, this seems to work fine for me?

brandond@dev01:~$ docker buildx create --bootstrap --name=k3s --driver=kubernetes
[+] Building 6.2s (1/1) FINISHED
 => [internal] booting buildkit                                                                                                                                                                                                                6.2s
 => => waiting for 1 pods to be ready, timeout: 2 minutes                                                                                                                                                                                      6.2s
k3s

brandond@dev01:~$ kubectl get pod -n default -o wide
NAME                    READY   STATUS    RESTARTS   AGE    IP          NODE                         NOMINATED NODE   READINESS GATES
k3s0-5446bf99dd-sg9bh   1/1     Running   0          103s   10.42.0.9   k3s-server-001.example.com   <none>           <none>

brandond@dev01:~$ kubectl get node -o wide
NAME                         STATUS   ROLES           AGE     VERSION        INTERNAL-IP   EXTERNAL-IP   OS-IMAGE           KERNEL-VERSION    CONTAINER-RUNTIME
k3s-server-001.example.com   Ready    control-plane   2m29s   v1.34.3+k3s1   172.17.0.4    <none>        K3s v1.34.3+k3s1   6.14.0-1012-aws   containerd://2.1.5-k3s1

[brandond]

Also you said you're on k3s version v1.34.2+k3s1 (8dac81b2) which still has runc v1.3.3.

Whatever's going on, I think this is a problem with your node, not with K3s or runc.

[rogue73]

Also you said you're on k3s version v1.34.2+k3s1 (8dac81b2) which still has runc v1.3.3.

Whatever's going on, I think this is a problem with your node, not with K3s or runc.

oh my fault. I picked the k3s -v output after I had rollback the snapshot.
With k3s 1.34.2 the problem does not exist and after upgrading to 1.34.3 I get the error above.

In my installation the creation of the builder pod is done by a gitea (git server) with gitea-actions and a gitea-runner pod in k3s.
In gitea actions i use the buildx-action https://github.com/docker/setup-buildx-action

I'll try to do a manual test also.