Docker containers inside sysbox container fail to start on new runc versions

[deansheather]

The newest runc version uses the rdma cgroup controller which isn't implemented in Sysbox 0.5.0

Inside of a sysbox container running docker and affected runc version:

$ docker run -it --rm hello-world
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to apply cgroup configuration: mkdir /sys/fs/cgroup/rdma/docker: permission denied: unknown.

$ runc --version
runc version 1.1.1
commit: v1.1.1-0-g52de29d
spec: 1.0.2-dev
go: go1.17.9
libseccomp: 2.5.1

$ sudo mkdir /sys/fs/cgroup/rdma/test
mkdir: cannot create directory '/sys/fs/cgroup/rdma/test': Permission denied

$ sudo mkdir /sys/fs/cgroup/cpu,cpuacct/test

$ ls -alh /sys/fs/cgroup
total 0
dr-xr-xr-x 14 root   root    360 May  6 21:17 .
drwxr-xr-x  9 nobody nogroup   0 May  6 21:17 ..
drwxr-xr-x  3 root   root      0 May  6 21:21 blkio
lrwxrwxrwx  1 root   root     11 May  6 21:17 cpu -> cpu,cpuacct
drwxr-xr-x  4 root   root      0 May  6 21:23 cpu,cpuacct
lrwxrwxrwx  1 root   root     11 May  6 21:17 cpuacct -> cpu,cpuacct
drwxr-xr-x  3 root   root      0 May  6 21:21 cpuset
drwxr-xr-x  3 root   root      0 May  6 21:21 devices
drwxr-xr-x  3 root   root      0 May  6 21:21 freezer
drwxr-xr-x  3 root   root      0 May  6 21:21 hugetlb
drwxr-xr-x  3 root   root      0 May  6 21:21 memory
lrwxrwxrwx  1 root   root     16 May  6 21:17 net_cls -> net_cls,net_prio
drwxr-xr-x  3 root   root      0 May  6 21:21 net_cls,net_prio
lrwxrwxrwx  1 root   root     16 May  6 21:17 net_prio -> net_cls,net_prio
drwxr-xr-x  3 root   root      0 May  6 21:21 perf_event
drwxr-xr-x  3 root   root      0 May  6 21:21 pids
dr-xr-xr-x  2 nobody nogroup   0 May  6 18:39 rdma
drwxr-xr-x  2 root   root      0 May  6 21:17 systemd

Notably, the rdma cgroup dir is owned by nobody:nogroup. Changing the permissions of /sys/fs/cgroup/rdma on the host to 777 allows containers to start correctly but is not an ideal solution.

[rodnymolina]

RDMA cgroup support was added to oci-runc a few months ago, but it hasn't been ported to Sysbox yet. This explains why problem is reproduced only with the very latest runc releases.

By porting and adjusting oci-runc's implementation we should be able to display rdma hierarchy with the proper file credentials.

[ctalledo]

FYI: I am working on fixing this right now; ETA end of week (05/14/22).

[rodnymolina]

As mentioned above, problem is only reproduced with the latest oci runc release (1.1.1), which is being packaged as part of the latest Docker version (20.10.15). Therefore, for Docker users, a workaround would be to downgrade to the 20.10.14 (or earlier) release.

[ctalledo]

Fix committed to Sysbox upstream via this PR. Will be present in upcoming v0.5.2 release.

Closing.

[aletundo]

Hi!

I was wondering whether this fix going to be released for the Community Edition too. The latest v0.5.2 release contains Enterprise binaries only.

Thanks!

[ctalledo]

I was wondering wether this fix going to be released for the Community Edition too. The latest v0.5.2 release contains Enterprise binaries only.

Hi @aletundo, this looks like a mistake on our part, thanks for pointing it out.

@rodnymolina: for some reason the sysbox-ee binaries are showing up on the sysbox community repo; could you take a look and upload the sysbox community binaries instead? Thanks!

[rodnymolina]

@aletundo, there was indeed an issue with our package links. It's already fixed now. Thanks for letting us know.

[aletundo]

@aletundo, there was indeed an issue with our package links. It's already fixed now. Thanks for letting us know.

You're welcome!