HostPID Pod Container Cgroup path was residual after container restarts

[Burning1020]

Description

We created a HostPID pod that has shared pid namespace with host. The container process was killed and then restarted again and agian. We found that the container cgroup path under /sys/fs/cgroup/<subsystem>/kubepods/podxxx-xxx/<contaienrID>/ was left.

The reason is that runc kill or runc delete did not really wait for the exit of container children process, p.wait() will receive ECHILD immediately, see https://github.com/opencontainers/runc/blob/v1.1.9/libcontainer/init_linux.go#L585C18-L585C18. If any child process is still running, the cgroup path couldn't be removed.

Steps to reproduce the issue

1. Create a HostPID pod, the container has many children process died and new born.
2. Kill the main container process for many times.
3. Container will be restart again by kubelet.

Describe the results you received and expected

Expected: The container cgroup path is deleted.
Received: Still exist.

What version of runc are you using?

runc version 1.1.9
commit: v1.1.9-0-gccaecfcb
spec: 1.0.2-dev
go: go1.20.3
libseccomp: 2.5.4

Host OS information

No response

Host kernel information

No response

[kolyshkin]

The code you referred to was recently changed in #3825.

In fact, runc kill or runc delete can't use wait(2) because those processes are not children of (this) runc invocation.

I think what happens is kubelet calls runc delete -f, and (in runc 1.1.x) it does not autodetect that this is a host pidns process and thus all the processes should be killed, not just init (this is solved in #3825 and will be available in runc 1.2).

The workaround is to call runc kill -a (for host pidns container) followed by runc delete, or wait till runc 1.2. One other option would be to backport #3825 to runc 1.1.x, but it's pretty large so I'd rather not do that.

[kolyshkin]

@Burning1020 If you can try with runc from main branch and check if it fixes your issue, that would be great!

[lifubang]

2. Kill the main container process for many times.

If we create a container with a shared or host PID namespace, after the init process has dead, the container's state becomes Stopped, so it will lead to ‘3. Container will be restart again by kubelet.’

Maybe we should trans this type container’s state to stopped by not only checking the init process has dead or not, but also checking whether there is no pid in the cgroup or not. Or just only the last condition.

[lifubang]

But I think Kubelet should delete the stopped container first and then start a new one. Did you see some error logs in k8s? Such as: ERRO[0000] Failed to remove paths: map[:/sys/fs/cgroup/unified/test blkio:/sys/fs/cgroup/blkio/user.slice/test cpu:/sys/fs/cgroup/cpu,cpuacct/user.slice/test cpuacct:/sys/fs/cgroup/cpu,cpuacct/user.slice/test cpuset:/sys/fs/cgroup/cpuset/test devices:/sys/fs/cgroup/devices/user.slice/test freezer:/sys/fs/cgroup/freezer/test hugetlb:/sys/fs/cgroup/hugetlb/test memory:/sys/fs/cgroup/memory/user.slice/user-1000.slice/session-8.scope/test misc:/sys/fs/cgroup/misc/test name=systemd:/sys/fs/cgroup/systemd/user.slice/user-1000.slice/session-8.scope/test net_cls:/sys/fs/cgroup/net_cls,net_prio/test net_prio:/sys/fs/cgroup/net_cls,net_prio/test perf_event:/sys/fs/cgroup/perf_event/test pids:/sys/fs/cgroup/pids/user.slice/user-1000.slice/session-8.scope/test rdma:/sys/fs/cgroup/rdma/test]

[lifubang]

and (in runc 1.1.x) it does not autodetect that this is a host pidns process and thus all the processes should be killed, not just init

I think it has detected:
https://github.com/opencontainers/runc/blob/v1.1.9/libcontainer/state_linux.go#L38-L44

[kolyshkin]

Maybe the source of the problem here is we treat the host pidns container with dead init as stopped, which is not quite true. So, a way to fix this is to modify runc to say the container is in running state once there are some processes left it its cgroup (and use runc kill -a (-a is not needed since runc 1.2) to actually kill those leftovers.

[Burning1020]

@Burning1020 If you can try with runc from main branch and check if it fixes your issue, that would be great!

@kolyshkin I have tried the main branch,

# runc --version
runc version 1.1.0+dev
commit: v1.1.0-791-g90cbd11
spec: 1.1.0+dev
go: go1.18.5
libseccomp: 2.5.0

The bug is not resolved!

[lifubang]

The bug is not resolved!

Sorry, there is a bug for shared pid namespace in the main branch. Please see #4047 .
Do you have a detailed step to reproduce your issue? If you can reproduce it with Docker, it will be more better.

[lifubang]

Do you have a detailed step to reproduce your issue?

For example, what is the container's image the pod uses? The content of the pod's yaml description file.

[Burning1020]

Do you have a detailed step to reproduce your issue?

For example, what is the container's image the pod uses? The content of the pod's yaml description file.

Reproduction is very simple.

1. Run a Pod under a Kubernetes cluster with HostPID=true, the container process should continuously fork child processes, you can use Kubernetes node-problem-detector for example.
2. Set the memory limit to a very small value, like 50Mb
3. The npd container was killed by OOM Killer for a few seconds and restarted by Kubelet but no pod was rebuilt.
4. Check the cgroup path

[lifubang]

I can't reproduce it in my test env. You mean the OOM killed container was deleted by kubelet, but the cgroup path still existed?