Launching container from image with significantly-sized VOLUME is slow

[struanb]

Inside a Sysbox container, launching a container from an image derived from a Dockerfile that includes a VOLUME directive (and where contains a 'significant' amount of data, in my test case ~300Mb) is unreasonably slow.

To reproduce:

1. Launch Sysbox container with embedded dockerd; launch dockerd; pull test image containing large VOLUME; time launching container from this image.

# docker run --rm -it --name=outer --entrypoint=/bin/bash --runtime=sysbox-runc newsnowlabs/dockside
root@236d9c17316c:/home/newsnow# dockerd >/var/log/dockerd.log 2>&1 &
[1] 61
root@236d9c17316c:/home/newsnow# docker pull newsnowlabs/dockside
Using default tag: latest
latest: Pulling from newsnowlabs/dockside
a024302f8a01: Pull complete 
df1da4b92cd3: Pull complete 
e4ae4a1e23f8: Pull complete 
4f4fb700ef54: Pull complete 
98f1c1c5c160: Pull complete 
293f2365040a: Pull complete 
862de65123f6: Pull complete 
b900f458021c: Pull complete 
9f1db669c557: Pull complete 
3841da2a0779: Pull complete 
e95cbfe01117: Pull complete 
8125781aea68: Pull complete 
7949e2b10877: Pull complete 
a68dbb2a997e: Pull complete 
0443950b5af6: Pull complete 
bfc030c66e65: Pull complete 
34f8d1b2b613: Pull complete 
1d8d1728d7a8: Pull complete 
e026fe1c0a9b: Pull complete 
928be91a56e1: Pull complete 
7a096dc4b3d6: Pull complete 
a9a1085491df: Pull complete 
Digest: sha256:dff454491fa252667e078819658b8fa8a6428283506bf79a7021adbac8290ee2
Status: Downloaded newer image for newsnowlabs/dockside:latest
docker.io/newsnowlabs/dockside:latest
root@236d9c17316c:/home/newsnow# time docker run --rm -it newsnowlabs/dockside >/dev/null

real	2m32.957s
user	0m0.162s
sys	0m0.068s

2. For comparison, time launching a container from the same image, but on the host:

# time docker run --rm -it newsnowlabs/dockside >/dev/null
real	0m3.801s
user	0m0.020s
sys	0m0.026s

[ctalledo]

Thanks @struanb for filing this issue, will take a look very soon.

[ctalledo]

Hi @struanb, I was able to repro the very severe slowdown you are seeing, and have root-caused it to Sysbox intercepting a couple of syscalls (fchown and llistxattr), both of which are used extensively when the inner Docker pulls & extracts the image, and when it copies the image's contents to a volume. If I remove the interception of those syscalls, the inner container image pull and start time are same as you would experience them at host level.

There is a reason Sysbox intercepts those syscalls inside the container, but it's usually to cover corner cases, so the cost of intercepting them may not be worthwhile. I am thinking how to solve this. One possibility is to make the interception of those syscalls configurable, with the default being to not intercept.

[ctalledo]

A solution for this has been merged and will be present in the upcoming v0.5.0 release.

The solution is to tell Sysbox to not intercept the *xattr() syscalls by passing the SYSBOX_ALLOW_TRUSTED_XATTR=FALSE env var to the container, as follows:

docker run --runtime=sysbox-runc -it --rm -e SYSBOX_ALLOW_TRUSTED_XATTR=FALSE <Sysbox-Container-Image>

Alternatively, one can ask Sysbox to do this for all containers by editing the sysbox-mgr systemd unit and adding the --allow-trusted-xattr=false flag to the sysbox-mgr's command line.

After doing this, the problem mentioned in this PR is resolved. For example, starting the newsnowlabs/dockside inside a Sysbox container now takes only a few seconds (similar to starting it at host level).

One caveat: the reason Sysbox intercepts the *xattr() syscalls in the first place is to allow the trusted.overlay.opaque xattr to be set from within the container's user-namespace (otherwise setting that attribute fails, which in turn causes older versions of Docker to sometimes fail when pulling images). Starting with Docker 20.10.9 and in hosts with kernel >= 5.11 this is not required (i.e., Docker uses a different attribute call user.overlay.opaque). Thus, if you want to use SYSBOX_ALLOW_TRUSTED_XATTR=FALSE, do so on hosts with kernel >= 5.11 and install the latest version of Docker inside the Sysbox container. Otherwise you run the risk of the inner Docker failing to pull images in some cases.

[ctalledo]

Note that this solution applies to the upcoming v0.5.0 release. It's not present in v0.4.1.

[ctalledo]

Closing.