"--use-new-run" problem with /dev/shm

[darkdragon-001]

Actual behavior
Loading some layers from cache fails.

Error extracting from cache: "removing whiteout .wh.dev: unlinkat //dev/shm: device or resource busy"

Expected behavior
Kaniko should load all layers from cache without problems.

To Reproduce
Kanik arguments: --use-new-run --cache=true
Steps to reproduce the behavior:

1. Build image with empty cache -> build success, pull failure (see comment below)
2. Build image loading from cache -> build failure

When removing the faulty layer from cache, it builds once. But when loading from cache the next time, it fails again.
I can also reproduce this when installing software-properties-common instead of sudo package.

Additional Information

• Dockerfile

FROM ubuntu:hirsute

RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -yq sudo && rm -rf /var/lib/apt/lists/*

• Build Context
  empty
• Kaniko Image
  gcr.io/kaniko-project/executor:debug

Kaniko is run from Gitlab CI in a Kubernetes runner.

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[darkdragon-001]

Update: When I try to docker pull the image to my local machine which was built successfully on the first run or after removing the faulty cache layer, it fails with

failed to register layer: Error processing tar file(exit status 1): link /dev /dev/shm: operation not permitted

---

Reproduce via:

IMAGE="kaniko-error:latest"
docker run -it --rm \     
      -v $(pwd):/workspace:rw \
          gcr.io/kaniko-project/executor:debug \
                --context dir:///workspace \
                --dockerfile Dockerfile \
                --destination "$IMAGE" \
                --use-new-run
docker pull $IMAGE

[jfo84]

We are seeing similar behavior, also on gitlab CI

[kamaln7]

We're also seeing a similar issue (only with --user-new-run), however the first build succeeds and the image works, but any subsequent builds that try to use the cache will error:

RUN apt-get update && apt-get install -y .... && rm -rf /var/lib/apt/lists/* 
INFO[0012] Found cached layer, extracting to filesystem 
error building image: error building stage: failed to execute command: extracting fs from image: removing whiteout .wh.dev: unlinkat //dev/pts/ptmx: operation not permitted

Digging a little more with --verbosity trace it seems to have added a whiteout for /dev (even though it is ignored by virtue of being a mounted dir), which causes Kaniko to try and delete /dev when extracting the cached layer.

---

First build, no cache

/dev is ignored

[37mTRAC�[0m[0018] Read the following line from /proc/self/mountinfo: 891 888 0:147 / /dev rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755 
�[37mTRAC�[0m[0018] Appending /dev from line: 891 888 0:147 / /dev rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755 

later in the logs:

[36mINFO�[0m[0023] RUN  apt-get .......
�[37mDEBU�[0m[0023] using new RunMarker command                  
�[37mTRAC�[0m[0023] Skipping paths under /kaniko, as it is a ignored directory 
�[37mTRAC�[0m[0023] Skipping paths under /dev, as it is a ignored directory 
�[37mTRAC�[0m[0023] Skipping paths under /proc, as it is a ignored directory 
�[37mTRAC�[0m[0023] Skipping paths under /sys, as it is a ignored directory 
�[37mTRAC�[0m[0023] Skipping paths under /workspace, as it is a ignored directory 
�[36mINFO�[0m[0023] cmd: /bin/sh
�[36mINFO�[0m[0023] args: [-c apt-get .......] 
�[36mINFO�[0m[0023] Running: [/bin/sh -c apt-get .......] 

....... apt output .......

�[37mTRAC�[0m[0047] Skipping paths under /kaniko, as it is a ignored directory 
�[37mTRAC�[0m[0048] Skipping paths under /dev, as it is a ignored directory 
�[37mTRAC�[0m[0048] Skipping paths under /proc, as it is a ignored directory 
�[37mTRAC�[0m[0048] Skipping paths under /sys, as it is a ignored directory 
�[37mTRAC�[0m[0048] Skipping paths under /workspace, as it is a ignored directory 
�[37mDEBU�[0m[0048] files changed [........]  # this is a huge list but there are no /dev paths in it
�[37mTRAC�[0m[0049] Skipping paths under /kaniko, as it is a ignored directory 
�[37mTRAC�[0m[0049] Skipping paths under /dev, as it is a ignored directory 
�[37mTRAC�[0m[0049] Skipping paths under /proc, as it is a ignored directory 
�[37mTRAC�[0m[0049] Skipping paths under /sys, as it is a ignored directory 
�[37mTRAC�[0m[0049] Skipping paths under /workspace, as it is a ignored directory 
�[37mDEBU�[0m[0049] Adding whiteout for /dev

Second build, reuses cache

......

�[36mINFO�[0m[0006] Taking snapshot of full filesystem...        
�[37mTRAC�[0m[0006] Skipping paths under /kaniko, as it is a ignored directory 
�[37mTRAC�[0m[0006] Skipping paths under /dev, as it is a ignored directory 
�[37mTRAC�[0m[0006] Skipping paths under /proc, as it is a ignored directory 
�[37mTRAC�[0m[0006] Skipping paths under /sys, as it is a ignored directory 
�[37mTRAC�[0m[0006] Skipping paths under /workspace, as it is a ignored directory 
�[37mTRAC�[0m[0006] Resolving paths [/ /.dockerenv /etc /etc/nsswitch.conf] 
�[36mINFO�[0m[0006] RUN  apt-get .... 
�[36mINFO�[0m[0006] Found cached layer, extracting to filesystem 

......

�[37mTRAC�[0m[0006] Read the following line from /proc/self/mountinfo: 891 888 0:147 / /dev rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755 
�[37mTRAC�[0m[0006] Appending /dev from line: 891 888 0:147 / /dev rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755 
�[37mTRAC�[0m[0006] Read the following line from /proc/self/mountinfo: 892 891 0:148 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666 

......

�[37mDEBU�[0m[0006] Mounted directories: [{/kaniko false} {/etc/mtab false} {/tmp/apt-key-gpghome true} {/var/run false} {/proc false} {/dev false} {/dev/pts false} {/sys false} {/sys/fs/cgroup false} {/dev/mqueue false} {/dev/shm false} {/workspace false} {/etc/resolv.conf false} {/etc/hostname false} {/etc/hosts false} {/kaniko/.docker/config.json false} {/dev/console false} {/proc/bus false} {/proc/fs false} {/proc/irq false} {/proc/sys false} {/proc/sysrq-trigger false} {/proc/acpi false} {/proc/kcore false} {/proc/keys false} {/proc/timer_list false} {/proc/sched_debug false} {/proc/scsi false} {/sys/firmware false}] 
�[37mTRAC�[0m[0006] Extracting layer 0 of media type application/vnd.docker.image.rootfs.diff.tar.gzip 
�[37mDEBU�[0m[0007] Whiting out /.wh.dev                         
error building image: error building stage: failed to execute command: extracting fs from image: removing whiteout .wh.dev: unlinkat //dev/console: device or resource busy