COPY to symlink destination behaviour differs from docker build

[LFrobeen]

I am trying to build an image containing a node app that manages dependencies using symlinks which seems to cause kaniko to fail during a COPY operation.

Stripping away any unrelated details, I am basically trying to build this image:

FROM alpine:latest as BASE

WORKDIR /usr/src/app

RUN mkdir -p packages/scripts
RUN echo hello world > packages/scripts/data

RUN mkdir node_modules
RUN ln -s ../packages/scripts /usr/src/app/node_modules/sourced-scripts

FROM alpine:latest as TARGET

WORKDIR /

COPY --from=BASE /usr/src/app/node_modules /app/node_modules
COPY --from=BASE /usr/src/app/packages/scripts /app/node_modules/sourced-scripts

RUN ls -l /app/node_modules
RUN cat /app/node_modules/sourced-scripts/data

Actual behavior
The build works as expected until it reaches the line
COPY --from=BASE /usr/src/app/packages/scripts /app/node_modules/sourced-scripts
where it should copy the scripts folder from the BASE stage into the target of the symlink /app/node_modules/sourced-scripts in the TARGET stage. However, it fails with this error instead:
building image: error building stage: failed to execute command: resolving dest symlink: failed to eval symlinks: lstat /app/packages: no such file or directory

The error indicates to me that kaniko is expecting the target path of the symlink to exist before the copy operation is executed.

Expected behavior
I would expect that kaniko would ignore the fact that the path of the symlink target does not exist yet and create it as part of the copy operation instead. This seems to be what docker build is doing for instance, as the dockerfile above works fine with standard docker.
If I am not mistaken, this is also the general semantics of the copy command, i.e. if the target path does not exist it should be created as needed.

To Reproduce
Steps to reproduce the behavior:

1. Build the above image with kaniko

Additional Information

• Dockerfile: see aboce
• Build Context: no files needed
• Kaniko Image: gcr.io/kaniko-project/executor@sha256:e61c08a27311884ff0d39a3ed4467f3c23444d73e03387fc9c7d5c6442796bae

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[drawks]

Any word on this? This behavior is mentioned in at least 3 issues, but none of them have any updates in them.

[imjasonh]

Any word on this? This behavior is mentioned in at least 3 issues, but none of them have any updates in them.

No updates at this time.

Kaniko is effectively staffed and maintained on a best-effort basis currently. That may change, but for now it's a pretty skeleton crew. I try to fix things when I can, and get notifications about issue comments like this, but I'm far from an expert, and don't have much time to dedicate to it.

If you're able to identify a fix, I'm happy to review and merge the PR (especially if it includes tests!!).

[guillaume-d]

The title of this bug is still misleading ("COPY to symlink [...]"), the problem here seems rather to be that the symlink did not get stored correctly in the previosu image as indeed in #1737, but also in #1944 (and maybe #437?).
And then of course the COPY to the missing symlink does not work.

[gabyx]

@drawks: Should be resolved in #2066
I added a rergression test in #2096 which highlights the troubles.