Multistage build removing build files and crashing

[plachta11b]

After repeated /kaniko/executor execution the Busybox commands (du, ls, etc..) start to be "not found". When constructing a minimal reproducible example I encountered possibly different error described below:

Actual behavior
🔥🔥🔥All files in the current working dir are removed as result!🔥🔥🔥 The multistage build ends up with an error.

INFO[2023-10-17T15:28:39Z] Resolved base name ghcr.io/grpc-ecosystem/grpc-health-probe:v0.4.18 to grpc_health_probe 
INFO[2023-10-17T15:28:39Z] Retrieving image manifest ghcr.io/grpc-ecosystem/grpc-health-probe:v0.4.18 
INFO[2023-10-17T15:28:39Z] Retrieving image ghcr.io/grpc-ecosystem/grpc-health-probe:v0.4.18 from registry ghcr.io 
INFO[2023-10-17T15:28:40Z] Retrieving image manifest mcr.microsoft.com/dotnet/aspnet:7.0 
INFO[2023-10-17T15:28:40Z] Retrieving image mcr.microsoft.com/dotnet/aspnet:7.0 from registry mcr.microsoft.com 
INFO[2023-10-17T15:28:43Z] Built cross stage deps: map[0:[/ko-app/grpc-health-probe]] 
INFO[2023-10-17T15:28:43Z] Retrieving image manifest ghcr.io/grpc-ecosystem/grpc-health-probe:v0.4.18 
INFO[2023-10-17T15:28:43Z] Returning cached image manifest              
INFO[2023-10-17T15:28:43Z] Executing 0 build triggers                   
INFO[2023-10-17T15:28:43Z] Building stage 'ghcr.io/grpc-ecosystem/grpc-health-probe:v0.4.18' [idx: '0', base-idx: '-1'] 
INFO[2023-10-17T15:28:48Z] Saving file ko-app/grpc-health-probe for later use 
INFO[2023-10-17T15:28:48Z] Deleting filesystem...                       
INFO[2023-10-17T15:28:48Z] Retrieving image manifest mcr.microsoft.com/dotnet/aspnet:7.0 
INFO[2023-10-17T15:28:48Z] Returning cached image manifest              
INFO[2023-10-17T15:28:48Z] Executing 0 build triggers                   
INFO[2023-10-17T15:28:48Z] Building stage 'mcr.microsoft.com/dotnet/aspnet:7.0' [idx: '1', base-idx: '-1'] 
INFO[2023-10-17T15:28:48Z] Unpacking rootfs as cmd COPY --from=grpc_health_probe /ko-app/grpc-health-probe /bin/grpc_health_probe requires it. 
INFO[2023-10-17T15:29:41Z] WORKDIR /app                                 
INFO[2023-10-17T15:29:41Z] Cmd: workdir                                 
INFO[2023-10-17T15:29:41Z] Changed working directory to /app            
INFO[2023-10-17T15:29:41Z] Creating directory /app with uid -1 and gid -1 
INFO[2023-10-17T15:29:41Z] Taking snapshot of files...                  
INFO[2023-10-17T15:29:42Z] COPY --from=grpc_health_probe /ko-app/grpc-health-probe /bin/grpc_health_probe 
INFO[2023-10-17T15:29:42Z] Taking snapshot of files...                  
error building image: error building stage: failed to get files used from context: failed to get fileinfo for /workspace: lstat /workspace: no such file or directory

Expected behavior

docker build . --build-arg ASPNET_IMAGE="mcr.microsoft.com/dotnet/aspnet:7.0" --build-arg GRPC_HEALTH_IMAGE="ghcr.io/grpc-ecosystem/grpc-health-probe:v0.4.18"

Sending build context to Docker daemon  2.048kB
Step 1/8 : ARG GRPC_HEALTH_IMAGE
Step 2/8 : ARG ASPNET_IMAGE
Step 3/8 : FROM $GRPC_HEALTH_IMAGE AS grpc_health_probe
v0.4.18: Pulling from grpc-ecosystem/grpc-health-probe
10f855b03c8a: Pull complete 
fe5ca62666f0: Pull complete 
fff4e558ad3a: Pull complete 
fcb6f6d2c998: Pull complete 
e8c73c638ae9: Pull complete 
1e3d9b7d1452: Pull complete 
4aa0ea1413d3: Pull complete 
7c881f9ab25e: Pull complete 
5627a970d25e: Pull complete 
250c06f7c38e: Pull complete 
9cf1101f96a4: Pull complete 
Digest: sha256:59101534d86dbd4f2e8fe86c42e7f40ac6be0c9f74089e7bdeeab82c21c6d26b
Status: Downloaded newer image for ghcr.io/grpc-ecosystem/grpc-health-probe:v0.4.18
 ---> 4d0362e4aaed
Step 4/8 : FROM $ASPNET_IMAGE
 ---> b06452c8ee91
Step 5/8 : WORKDIR /app
 ---> Running in 8884715cef70
Removing intermediate container 8884715cef70
 ---> 59a86db52dae
Step 6/8 : COPY --from=grpc_health_probe /ko-app/grpc-health-probe /bin/grpc_health_probe
 ---> 74fc54e53935
Step 7/8 : COPY . .
 ---> ad5e6786fbf9
Step 8/8 : ENTRYPOINT ["echo", "hello world"]
 ---> Running in 7a67cc51f12e
Removing intermediate container 7a67cc51f12e
 ---> a5fc183260e7
Successfully built a5fc183260e7

docker run a5fc183260e7

hello world

docker run --entrypoint="" a5fc183260e7 ls -al /bin/grpc_health_probe

-r-xr-xr-x 1 root root 11378915 Jan  1  1970 /bin/grpc_health_probe

docker run --entrypoint="" a5fc183260e7 ls -al

total 12
drwxr-xr-x 1 root root 4096 Oct 17 14:58 .
drwxr-xr-x 1 root root 4096 Oct 17 15:45 ..
-rw-rw-r-- 1 root root  240 Oct 17 14:57 Dockerfile

To Reproduce
Steps to reproduce the behavior:

1. docker run -it --entrypoint="" gcr.io/kaniko-project/executor:debug /bin/sh
2. vi Dockerfile # type 'i'; paste Dockerfile content; hit esc; type :wq; hit enter
3. run command mentioned in Build Context section

Additional Information - minimal reproducible example

• Build passing. Removes files in build dir.

FROM ghcr.io/grpc-ecosystem/grpc-health-probe:v0.4.18 AS grpc_health_probe
FROM mcr.microsoft.com/dotnet/aspnet:7.0
COPY --from=grpc_health_probe /ko-app/grpc-health-probe /bin/grpc_health_probe

ENTRYPOINT ["echo", "hello world"]

• Build Context - Only Dockerfile is needed to see files disapper

 /kaniko/executor \
    --log-timestamp=true \
    --dockerfile "Dockerfile" \
    --no-push

Additional Information - expected behaviour example

• Dockerfile used in Docker build

ARG GRPC_HEALTH_IMAGE
ARG ASPNET_IMAGE
FROM $GRPC_HEALTH_IMAGE AS grpc_health_probe
FROM $ASPNET_IMAGE
WORKDIR /app
COPY --from=grpc_health_probe /ko-app/grpc-health-probe /bin/grpc_health_probe
COPY . .

ENTRYPOINT ["echo", "hello world"]

• Dockerfile where build fails. Removes files in build dir.

FROM ghcr.io/grpc-ecosystem/grpc-health-probe:v0.4.18 AS grpc_health_probe
FROM mcr.microsoft.com/dotnet/aspnet:7.0
WORKDIR /app
COPY --from=grpc_health_probe /ko-app/grpc-health-probe /bin/grpc_health_probe
COPY . .

ENTRYPOINT ["echo", "hello world"]

• Build Context - Command with flags and args to replicate Docker expected behaviour

/kaniko/executor \
    --use-new-run=true \
    --log-timestamp=true \
    --build-arg ASPNET_IMAGE="mcr.microsoft.com/dotnet/aspnet:7.0" \
    --build-arg GRPC_HEALTH_IMAGE="ghcr.io/grpc-ecosystem/grpc-health-probe:v0.4.18" \
    --dockerfile "Dockerfile" \
    --no-push

• Kaniko Image (fully qualified with digest)
  gcr.io/kaniko-project/executor@sha256:b8c0977f88f24dbd7cbc2ffe5c5f824c410ccd0952a72cc066efc4b6dfbb52b6 Kaniko version : v1.16.0

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag (not exclusively)
Please check if your dockerfile is a multistage dockerfile

[plachta11b]

Probably duplicate of the #2764

[aaron-prindle]

@plachta11b thank you for flagging this issue and the detailed repro information you provided - as noted this likely a dupe of #2764. Going to close this for now and track this issue there, feel free to continue the thread there. Thanks!