Kaniko build fails to unpack rootfs containing docker mounts

[davidmccormick]

Actual behavior
When a build step requires that kaniko unpacks the rootfs when kaniko is running from within Docker, and we have mounted files within the rootfs filessystem, we get a failure. Our CI/CD pipeline always tries to mount in some additional commands.

Test Dockerfile: -

ARG from_image 
FROM ${from_image}

COPY tls/* /usr/local/share/ca-certificates/mycorp/

test_script.sh: -

#!/bin/bash

echo "test_script ran $@"

and build command...

docker run -it --rm \
      -v ${PWD}:/workspace:rw \
      -v ${PWD}/test_script.sh:/sbin/docker-init:ro \
	  gcr.io/kaniko-project/executor:v1.0.0 \
               --context dir:///workspace \
		--dockerfile Dockerfile \
		--build-arg from_image=kindest/node:v1.17.5 \
		--destination "output:latest" \
		--no-push

Resultant output: -

INFO[0000] Retrieving image manifest kindest/node:v1.17.5
INFO[0000] Retrieving image kindest/node:v1.17.5
INFO[0001] Retrieving image manifest kindest/node:v1.17.5
INFO[0001] Retrieving image kindest/node:v1.17.5
INFO[0004] Built cross stage deps: map[]
INFO[0004] Retrieving image manifest kindest/node:v1.17.5
INFO[0004] Retrieving image kindest/node:v1.17.5
INFO[0005] Retrieving image manifest kindest/node:v1.17.5
INFO[0005] Retrieving image kindest/node:v1.17.5
INFO[0007] Executing 0 build triggers
INFO[0007] Unpacking rootfs as cmd COPY tls/* /usr/local/share/ca-certificates/mycorp/ requires it.
error building image: error building stage: failed to get filesystem from image: error removing sbin to make way for new symlink: unlinkat //sbin/docker-init: device or resource busy

Expected behavior
I would like the build to automatically skip over the mounted in file when making its snapshot or give me a command line option of paths to skip when taking snapshots.

To Reproduce
Steps to reproduce the behavior:

1. Implement the Dockerfile, script and run the build as above.

Additional Information

• Dockerfile
  See above
• Build Context
  A tls directory containing one or more files.
• Kaniko Image gcr.io/kaniko-project/executor:v1.0.0 sha256:50388657b978fd417fef3fe0179ca507031c6b02ecb60973d776ce40ec864255

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[davidmccormick]

looks similar to #946

[davidmccormick]

In the source image /sbin is a symlink - as are other typical directories

dave@dmcc ~ % docker run -it --entrypoint=/bin/sh kindest/node:v1.17.5
# ls -al
total 64
drwxr-xr-x   1 root root 4096 Sep 14 16:31 .
drwxr-xr-x   1 root root 4096 Sep 14 16:31 ..
-rwxr-xr-x   1 root root    0 Sep 14 16:31 .dockerenv
lrwxrwxrwx   1 root root    7 Apr 10 20:34 bin -> usr/bin
drwxr-xr-x   2 root root 4096 Oct 14  2019 boot
drwxr-xr-x   2 root root 4096 May  1 02:12 build
drwxr-xr-x   5 root root  360 Sep 14 16:31 dev
drwxr-xr-x   1 root root 4096 Sep 14 16:31 etc
drwxr-xr-x   2 root root 4096 Oct 14  2019 home
drwxr-xr-x   5 root root 4096 May  1 02:12 kind
lrwxrwxrwx   1 root root    7 Apr 10 20:34 lib -> usr/lib
lrwxrwxrwx   1 root root    9 Apr 10 20:34 lib32 -> usr/lib32
lrwxrwxrwx   1 root root    9 Apr 10 20:34 lib64 -> usr/lib64
lrwxrwxrwx   1 root root   10 Apr 10 20:34 libx32 -> usr/libx32
drwxr-xr-x   2 root root 4096 Apr 10 20:34 media
drwxr-xr-x   2 root root 4096 Apr 10 20:34 mnt
drwxr-xr-x   1 root root 4096 May  1 02:12 opt
dr-xr-xr-x 225 root root    0 Sep 14 16:31 proc
drwx------   2 root root 4096 Apr 10 20:35 root
drwxr-xr-x   1 root root 4096 May  1 02:12 run
lrwxrwxrwx   1 root root    8 Apr 10 20:34 sbin -> usr/sbin
drwxr-xr-x   2 root root 4096 Apr 10 20:34 srv
dr-xr-xr-x  12 root root    0 Sep 14 15:48 sys
drwxrwxrwt   2 root root 4096 Apr 10 20:35 tmp
drwxr-xr-x   1 root root 4096 Apr 30 20:13 usr
drwxr-xr-x   1 root root 4096 Apr 10 20:35 var

[davidmccormick]

The issue looks to be more general, the kindest/node image is built from ubuntu:20.10
Using the following to simulate the docker 19.x behaviour which mounts docker-init into '/sbin'

docker run -it --rm \
      -v ${PWD}:/workspace:rw \
      -v ${PWD}/test_script.sh:/sbin/docker-init:ro \
	  gcr.io/kaniko-project/executor:v1.0.0 \
               --context dir:///workspace \
		--dockerfile Dockerfile \
		--build-arg from_image=ubuntu:20.10 \
		--destination "output:latest" \
		--no-push

I get the same error when kaniko tries to unpack the root image: -

dave@dmcc dave % ./build_kaniko.sh
INFO[0000] Retrieving image manifest ubuntu:20.10
INFO[0000] Retrieving image ubuntu:20.10
INFO[0001] Retrieving image manifest ubuntu:20.10
INFO[0001] Retrieving image ubuntu:20.10
INFO[0005] Built cross stage deps: map[]
INFO[0005] Retrieving image manifest ubuntu:20.10
INFO[0005] Retrieving image ubuntu:20.10
INFO[0006] Retrieving image manifest ubuntu:20.10
INFO[0006] Retrieving image ubuntu:20.10
INFO[0009] Executing 0 build triggers
INFO[0009] Unpacking rootfs as cmd RUN rm -f /sbin /bin requires it.
error building image: error building stage: failed to get filesystem from image: error removing sbin to make way for new symlink: unlinkat //sbin/docker-init: device or resource busy

Looking at the code in pkg/util/fs_util.go it looks as though function: -

func ExtractFile(dest string, hdr *tar.Header, tr io.Reader) error {
...
	case tar.TypeSymlink:
		logrus.Tracef("symlink from %s to %s", hdr.Linkname, path)
                ...
		// Check if something already exists at path
		// If so, delete it
		if FilepathExists(path) {
			if err := os.RemoveAll(path); err != nil {
				return errors.Wrapf(err, "error removing %s to make way for new symlink", hdr.Name)
			}
		}

is reading the root tar archives entries and upon finding a symlink it wants to replace the existing /sbin which is a real directory with the symlink in the rootfs of the image - where upon it hits the issue that docker 19.03 has mounted docker-init into the the pre-existing /sbin directory (so it can't be removed). This wasn't a problem with docker 18.03 which mounted init into /dev/init.

Unless I'm mistaken any container built from ubuntu:20.10 or which has a symlink in the place of `/sbin' is going to suffer from the same issue building with kaniko and docker 19.03.