Strange behavior while building a Kaniko image using Kaniko

[alexiri]

Actual behavior
Using Kaniko, I'm trying to build an image based on the upstream Kaniko image but including some extra tools. I can include these extra tools via COPY, but if I try to modify files via RUN, whatever changes I make don't appear in the resulting image.

Expected behavior
I expect changes performed via RUN to have an effect.

To Reproduce
Steps to reproduce the behavior:

1. Create a Dockerfile like this one:

FROM alpine:3.10 AS builder
RUN date > /this_works

FROM gcr.io/kaniko-project/executor:v1.8.1-debug

# This works
COPY --from=builder /this_works /kaniko/

# This seems to work (output looks fine during the build),
# but this file is missing from the resulting image
RUN date > /kaniko/this_does_not_work
RUN ls -l /kaniko/

2. Build the image using Kaniko:

docker run --rm -v $(pwd):/workspace -v $(pwd)/config.json:/kaniko/.docker/config.json:ro gcr.io/kaniko-project/executor:v1.8.1-debug --dockerfile /workspace/Dockerfile --destination alexiri/kaniko-test

3. Run the resulting image and look for both files in /kaniko:

docker run --rm -it --entrypoint /bin/sh alexiri/kaniko-test:latest -c "ls -l /kaniko"
latest: Pulling from alexiri/kaniko-test
...
Digest: sha256:f00b4360fd17fa77051093655afb2233ebf600df09d81a06c6195a6b1f97a258
Status: Downloaded newer image for alexiri/kaniko-test:latest
docker.io/alexiri/kaniko-test:latest
total 75236
-rwxr-xr-x    1 0        0         10890899 Mar 31 20:47 docker-credential-acr-env
-rwxr-xr-x    1 0        0          8980342 Mar 31 20:47 docker-credential-ecr-login
-rwxr-xr-x    1 0        0          7817536 Mar 31 20:46 docker-credential-gcr
-rwxr-xr-x    1 0        0         35061760 Apr  5 06:07 executor
drwxr-xr-x    3 0        0             4096 Apr  5 06:07 ssl
-rw-r--r--    1 0        0               29 Jun 25 13:19 this_works
-rwxr-xr-x    1 0        0         14278656 Apr  5 06:07 warmer

Additional Information

• Dockerfile
  Please provide either the Dockerfile you're trying to build or one that can reproduce this error.
• Build Context
  Please provide or clearly describe any files needed to build the Dockerfile (ADD/COPY commands)
• Kaniko Image (fully qualified with digest)

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[hown3d]

This is an issue that's not tight to the kaniko image itself.

Because kaniko uses the /kaniko directory as the directory to build the container image, nothing inside /kaniko will be appended to the resulting image.
You can use the --kaniko-dir flag to specify another base directory for the kaniko build.

Dockerfile for reproduction

FROM ubuntu

RUN date > /kaniko/this_is_date

kaniko run command

$ docker run -v $(pwd)/Dockerfile:/workspace/Dockerfile gcr.io/kaniko-project/executor -d image-ref:latest
INFO[0001] Retrieving image manifest ubuntu
INFO[0001] Retrieving image ubuntu from registry index.docker.io
INFO[0002] Built cross stage deps: map[]
INFO[0002] Retrieving image manifest ubuntu
INFO[0002] Returning cached image manifest
INFO[0002] Executing 0 build triggers
INFO[0002] Unpacking rootfs as cmd RUN date > /kaniko/this_is_date requires it.
INFO[0005] RUN date > /kaniko/this_is_date
INFO[0005] Taking snapshot of full filesystem...
INFO[0006] cmd: /bin/sh
INFO[0006] args: [-c date > /kaniko/this_is_date]
INFO[0006] Running: [/bin/sh -c date > /kaniko/this_is_date]
INFO[0006] Taking snapshot of full filesystem...
INFO[0006] No files were changed, appending empty layer to config. No layer added to image.
INFO[0006] Pushing image to image-ref:latest
INFO[0048] Pushed image-ref@sha256:blabla

Running built container image

$ docker run -ti image-ref:latest
root@c4c636fb003e:/# ls -al /kaniko
ls: cannot access '/kaniko': No such file or directory

I think this is a common issue that should be avoided.
@imjasonh maybe using a random temporary directory for builds and executing chroot would fix these issues? This would also fix #2153.

[imjasonh]

Thanks @hown3d for that (correct!) explanation. I'm definitely open to ideas about how to let Kaniko build itself more correctly. I just don't have any time to dedicate to writing code for it, but I'll happily review any PR that includes passing tests. 😄