can't start nginx image with runc and user namespaces (works with crun)

[rata]

Description

Trying to start a Kubernetes container with userns using the nginx official image, fails. This was reported here: containerd/containerd#10598 by @ctrox.

@ctrox also found a workaround: add "tty: true" to the kubernetes pod makes it work.

And a simpler repro: just a container with userns that runs "cat /dev/stderr" also fails with permission denied.

I guess you need to run detached (as containerd does) to hit this, otherwise it uses your shell and that probably works.

@ctrox thanks for the great bug report!

Sorry the brevity, I'm sick ATM. I'll add more info when I recover

Steps to reproduce the issue

No response

Describe the results you received and expected

Works, as without user namespaces.

What version of runc are you using?

runc 1.2.0

Host OS information

No response

Host kernel information

No response

[dims]

xref: kubernetes/kubernetes#128276

[kolyshkin]

@ctrox could you please try v1.2.0 + PR #4477? If you need a static binary, you can go to that PR, click on Checks tab, click on 'validate' in the left column, scroll all the way down and you'll see a tarball with binaries under "Artifacts".

[lifubang]

Could you confirm that it can works with crun?
I have a test case to reproduce it, but it also can't works with crun.

root@iZj6cgggwb62cxurec74geZ:/opt/bb# crun/crun run -d test
cat: can't open '/dev/stderr': Permission denied
root@iZj6cgggwb62cxurec74geZ:/opt/bb# crun/crun delete test
root@iZj6cgggwb62cxurec74geZ:/opt/bb# /root/go/src/github.com/opencontainers/runc/runc run -d test
cat: can't open '/dev/stderr': Permission denied
root@iZj6cgggwb62cxurec74geZ:/opt/bb# /root/go/src/github.com/opencontainers/runc/runc delete test
root@iZj6cgggwb62cxurec74geZ:/opt/bb# 

Maybe the core reason is that the permission of the /proc/self/fd/2 is 300? It should be 700 in the host. But I can't confirm.
The content of ls /proc/self/fd in the container looks like:

   dr-x------    2 root     root           4 Oct 26 01:10 .
   dr-xr-xr-x    9 root     root           0 Oct 26 01:10 ..
   lr-x------    1 root     root          64 Oct 26 01:10 0 -> pipe:[2264321]
   l-wx------    1 root     root          64 Oct 26 01:10 1 -> pipe:[2264322]
   l-wx------    1 root     root          64 Oct 26 01:10 2 -> pipe:[2264323]
   lr-x------    1 root     root          64 Oct 26 01:10 3
   ls: /proc/self/fd/3: cannot read link: No such file or directory

But in the host:

dr-x------ 2 root root  4 Oct 26 09:28 .
dr-xr-xr-x 9 root root  0 Oct 26 09:28 ..
lrwx------ 1 root root 64 Oct 26 09:28 0 -> /dev/pts/0
lrwx------ 1 root root 64 Oct 26 09:28 1 -> /dev/pts/0
lrwx------ 1 root root 64 Oct 26 09:28 2 -> /dev/pts/0
lr-x------ 1 root root 64 Oct 26 09:28 3 -> /proc/629444/fd

The content of config.json:

{ "ociVersion": "1.0.2-dev", "process": { "terminal": false, "user": { "uid": 0, "gid": 0 }, "args": [ "cat", "/dev/stderr" ], "env": [ "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "TERM=xterm" ], "cwd": "/", "capabilities": { "bounding": [ "CAP_AUDIT_WRITE", "CAP_KILL", "CAP_NET_BIND_SERVICE" ], "effective": [ "CAP_AUDIT_WRITE", "CAP_KILL", "CAP_NET_BIND_SERVICE" ], "permitted": [ "CAP_AUDIT_WRITE", "CAP_KILL", "CAP_NET_BIND_SERVICE" ], "ambient": [ "CAP_AUDIT_WRITE", "CAP_KILL", "CAP_NET_BIND_SERVICE" ] }, "rlimits": [ { "type": "RLIMIT_NOFILE", "hard": 1024, "soft": 1024 } ], "noNewPrivileges": true }, "root": { "path": "rootfs", "readonly": true }, "hostname": "runc", "mounts": [ { "destination": "/proc", "type": "proc", "source": "proc" }, { "destination": "/dev", "type": "tmpfs", "source": "tmpfs", "options": [ "nosuid", "strictatime", "mode=755", "size=65536k" ] }, { "destination": "/dev/pts", "type": "devpts", "source": "devpts", "options": [ "nosuid", "noexec", "newinstance", "ptmxmode=0666", "mode=0620", "gid=5" ] }, { "destination": "/dev/shm", "type": "tmpfs", "source": "shm", "options": [ "nosuid", "noexec", "nodev", "mode=1777", "size=65536k" ] }, { "destination": "/dev/mqueue", "type": "mqueue", "source": "mqueue", "options": [ "nosuid", "noexec", "nodev" ] }, { "destination": "/sys", "type": "sysfs", "source": "sysfs", "options": [ "nosuid", "noexec", "nodev", "ro" ] }, { "destination": "/sys/fs/cgroup", "type": "cgroup", "source": "cgroup", "options": [ "nosuid", "noexec", "nodev", "relatime", "ro" ] } ], "linux": { "resources": { "devices": [ { "allow": false, "access": "rwm" } ] }, "namespaces": [ { "type": "pid" }, { "type": "network" }, { "type": "ipc" }, { "type": "uts" }, { "type": "mount" }, { "type": "cgroup" }, { "type": "user" } ], "uidMappings": [{"hostID": 100000, "containerID": 0, "size": 65534}], "gidMappings": [{"hostID": 100000, "containerID": 0, "size": 65534}], "maskedPaths": [ "/proc/acpi", "/proc/asound", "/proc/kcore", "/proc/keys", "/proc/latency_stats", "/proc/timer_list", "/proc/timer_stats", "/proc/sched_debug", "/sys/firmware", "/proc/scsi" ], "readonlyPaths": [ "/proc/bus", "/proc/fs", "/proc/irq", "/proc/sys", "/proc/sysrq-trigger" ] } }

[lifubang]

Maybe the core reason is that the permission of the /proc/self/fd/2 is 300? It should be 700 in the host. But I can't confirm.

This is not the real reason, the core reason is that the permission of /proc/self/fd/0-2 in the host is 0o700, they can't be accessed by other user, we should change it to 0o666.

[lifubang]

@ctrox could you please try v1.2.0 + PR #4477?

It still have the issue.
@ctrox @rata Could you help to see if #4478 has fixed your issue or not?

[kolyshkin]

I think we need to go back to square one with this.

What we know now is

• there's a scenario where kubernetes v1.30 and containerd are used, and a process inside a container can't write to a file which is a symlink to /dev/stderr when runc-1.2.0-rc.3 (or rc.2 -- it's not entirely clear) is used;
• this happens if terminal is disabled, userns is enabled, and a non-root user inside a container;
• the same setup apparently works with crun;
• the reporter (@ctrox) was unable to reproduce the issue without kubernetes (only using containerd and crictl);
• neither containerd nor kubernetes have a test case for this issue.

What we don't know:

• were anybody else able to reproduce this;
• how to reproduce this without kubernetes (and, ideally, without containerd);
• whether some older version of runc works.

I think we need to have some sort of a reproducer first.

[cyphar]

It should also be noted that file passthrough for stdio is something folks should avoid using if possible. It's only really useful if you are using runc directly from shell scripts and don't want to use runc as a stdio forwarder (maybe you want to give a container a proper terminal or a network socket as stdio).

Within the context of containerd and Kubernetes it makes very little sense to use this mode. Using consoles improves container isolation and all of these higher-level tools have built-in support for handling consoles.