Missing symlink in the images built using Kaniko #1944

prasadkris · 2022-02-18T05:53:57Z

Actual behavior

I am attempting to migrate the image build process for our micro services from docker to kaniko and have realized that the kaniko built images fails to create files/symlinks generated by the OS packages.

Expected behavior

The image built from a Dockerfile using kaniko should the same as the ones built using Docker, it must include all files including symlinks.

To Reproduce
Steps to reproduce the behavior:

Use the following Dockerfile and build an image using kaniko and another one using docker, push both images to repo.

FROM python:3.10.2-slim

RUN apt-get update && apt-get install -y \
  libglib2.0-0 \
  libgl1-mesa-glx \
  && rm -rf /var/lib/apt/lists/*

Try running the images to experience the difference. The package libxdmcp6:amd64 generates a symlink /usr/lib/x86_64-linux-gnu/libXdmcp.so.6 pointing to /usr/lib/x86_64-linux-gnu/libXdmcp.so.6.0.0 which is missing in the image built using Kaniko and it does exist in the image built using Docker as shown below.

Kaniko

docker run --rm -it prasadzone/pythonapp:kaniko bash
root@3f1505cd4cee:/# dpkg -L libxdmcp6:amd64
/.
/usr
/usr/lib
/usr/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu/libXdmcp.so.6.0.0
/usr/share
/usr/share/doc
/usr/share/doc/libxdmcp6
/usr/share/doc/libxdmcp6/changelog.Debian.gz
/usr/share/doc/libxdmcp6/changelog.gz
/usr/share/doc/libxdmcp6/copyright
/usr/lib/x86_64-linux-gnu/libXdmcp.so.6
root@3f1505cd4cee:/# ls -la /usr/lib/x86_64-linux-gnu | grep libXdmcp
-rw-r--r--  1 root root    22728 Mar  2  2017 libXdmcp.so.6.0.0
root@3f1505cd4cee:/#

Docker

$ docker run --rm -it prasadzone/pythonapp:docker bash
root@864bf398734b:/# dpkg -L libxdmcp6:amd64
/.
/usr
/usr/lib
/usr/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu/libXdmcp.so.6.0.0
/usr/share
/usr/share/doc
/usr/share/doc/libxdmcp6
/usr/share/doc/libxdmcp6/changelog.Debian.gz
/usr/share/doc/libxdmcp6/changelog.gz
/usr/share/doc/libxdmcp6/copyright
/usr/lib/x86_64-linux-gnu/libXdmcp.so.6
root@864bf398734b:/# ls -la /usr/lib/x86_64-linux-gnu | grep libXdmcp
lrwxrwxrwx  1 root root       17 Mar  2  2017 libXdmcp.so.6 -> libXdmcp.so.6.0.0
-rw-r--r--  1 root root    22728 Mar  2  2017 libXdmcp.so.6.0.0

This deviation from docker image is a show stopper as the applications depending on this library will fail to start if we use kaniko based images (will get errors similar to below), the app works fine if we use the image built with docker.

  File "/usr/local/lib/python3.10/site-packages/cv2/__init__.py", line 8, in <module>
    from .cv2 import *
ImportError: libXdmcp.so.6: cannot open shared object file: No such file or directory

Additional Information

Dockerfile (Please refer to the reproduce steps)

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use `--cache` flag
Please check if your dockerfile is a multistage dockerfile

The text was updated successfully, but these errors were encountered:

prasus · 2022-03-08T08:58:36Z

Any updates/comments? 🙂

We were on the verge of replacing the dind containers (which builds the images in our CI) with Kaniko, but these deviations from docker image is really disappointing! 🙁

imjasonh · 2022-03-08T14:42:37Z

Any updates/comments? 🙂

We were on the verge of replacing the dind containers (which builds the images in our CI) with Kaniko, but these deviations from docker image is really disappointing! 🙁

Nope. Nobody is currently working on this issue. If you have any ideas about a fix, feel free to send a PR and I'll review it. If you (or anyone) would be interested in meeting over video some time and going through the code, I'd be willing to arrange that.

gabyx · 2022-03-17T17:54:46Z

I experience somehow the same weird issue. During a huge build, suddenly
libbsd0.so symlink is missing eventough it was installed by apt-get install libbsd0

gabyx · 2022-03-17T17:57:12Z

@imjasonh Any idea where the error could be located?

Just stumbled over this:
https://github.com/GoogleContainerTools/kaniko/blob/main/pkg/filesystem/resolve.go
Could this function return wrong stuff?

gabyx · 2022-03-17T18:07:32Z

I think that is super important, to get fixed, since this holds people back from using kaniko...

gabyx · 2022-03-17T18:26:59Z

I cannot reproduce this error with the current gcr.io/kaniko-project/executor:aca3b966a3376c998f4824162912ba5ea04c9d4f-debug. But the bug is there, as I've seen it in my build.

gabyx · 2022-03-17T19:34:36Z

I have the following output which shows the problem over two stages

INFO[0504] Taking snapshot of full filesystem...        
INFO[0513] RUN ls -al /usr/lib/x86_64-linux-gnu/libbsd.so.0 
INFO[0513] cmd: /bin/sh                                 
INFO[0513] args: [-c ls -al /usr/lib/x86_64-linux-gnu/libbsd.so.0] 
INFO[0513] util.Lookup returned: &{Uid:1000 Gid:1000 Username:developer Name: HomeDir:/home/developer} 
INFO[0513] performing slow lookup of group ids for developer 
INFO[0513] Running: [/bin/sh -c ls -al /usr/lib/x86_64-linux-gnu/libbsd.so.0] 
lrwxrwxrwx 1 root root 16 Aug  8  2019 /usr/lib/x86_64-linux-gnu/libbsd.so.0 -> libbsd.so.0.10.0
INFO[0513] Taking snapshot of full filesystem...        
INFO[0515] No files were changed, appending empty layer to config. No layer added to image. 
INFO[0515] WORKDIR "/home/$USER_NAME"                   
INFO[0515] cmd: workdir                                 
INFO[0515] Changed working directory to /home/developer 
INFO[0515] No files changed in this command, skipping snapshotting. 
INFO[0515] Storing source image from stage 0 at path /kaniko/stages/0 
INFO[0532] Deleting filesystem...                       

----------------------------------- HERE IS THE FIRST STAGE FINISHED (all good)

INFO[0540] Base image from previous stage 0 found, using saved tar at path /kaniko/stages/0 
INFO[0540] Executing 0 build triggers                   
INFO[0543] Unpacking rootfs as cmd RUN ls -al /usr/lib/x86_64-linux-gnu/libbsd.so.0 requires it. 
INFO[0572] ARG SECRETS_DIR                              
INFO[0572] ARG BUILD_DIR                                
INFO[0572] ARG USER_NAME                                
INFO[0572] ARG PYTHON_ENV                               
INFO[0572] ARG BUILD_VERSION                            
INFO[0572] ENV CONTAINER_BUILD_VERSION=$BUILD_VERSION   
INFO[0572] RUN ls -al /usr/lib/x86_64-linux-gnu/libbsd.so.0 
INFO[0572] Taking snapshot of full filesystem...        
INFO[0580] cmd: /bin/sh                                 
INFO[0580] args: [-c ls -al /usr/lib/x86_64-linux-gnu/libbsd.so.0] 
INFO[0580] util.Lookup returned: &{Uid:1000 Gid:1000 Username:developer Name: HomeDir:/home/developer} 
INFO[0580] performing slow lookup of group ids for developer 
INFO[0580] Running: [/bin/sh -c ls -al /usr/lib/x86_64-linux-gnu/libbsd.so.0] 
ls: cannot access '/usr/lib/x86_64-linux-gnu/libbsd.so.0': No such file or directory

----------------------------------- Now suddenly the symlink is missing.

It seems that the next initial state does not unpack symlinks properly (???)
This happens over the following stage from AAAA to BBBB

### FIRST STAGE
FROM ... as AAAA

RUN ls -al /usr/lib/x86_64-linux-gnu/libbsd.so.0

WORKDIR "/home/$USER_NAME"

### SECOND STAGE *until here all good*
FROM AAAA as BBBB
ARG SECRETS_DIR
ARG BUILD_DIR
ARG USER_NAME
ARG PYTHON_ENV
ENV CONTAINER_ID="7fff7106-af66-4e23-bba8-2278408b982c""

RUN ls -al /usr/lib/x86_64-linux-gnu/libbsd.so.0

Somehow I cannot find an MWE so far which shows the problem...

gabyx · 2022-03-18T13:28:15Z

@imjasonh : Could we set up a video call. That would be great.

DS-KrzJon · 2022-04-06T16:20:01Z

We have the same issue in our Pipeline with Jenkins and Kaniko in Version
kaniko-project/executor:aca3b966a3376c998f4824162912ba5ea04c9d4f

gabyx · 2022-04-28T06:11:47Z

@imjasonh: This ticket should be reopened: #2057 was probably only the tip of the iceberg of the problem. Sadly we still see this issue and #2057 does not solve it. We try to dig deeper.

gabyx · 2022-04-28T06:18:28Z

@imjasonh: Do you might have time next week for a video call or any other Kaniko Leada Dev? That would e great.

imjasonh · 2022-04-28T06:53:03Z

@imjasonh: Do you might have time next week for a video call or any other Kaniko Leada Dev? That would e great.

Sure, ping me on the Kubernetes Slack and we can set up a time. I'm pretty free Monday. You should also join #kaniko if you aren't already in there.

gabyx · 2022-04-28T07:29:09Z

See: #2066
Sure my initial PR was wrong, because I did not test it... damm...
Anyway, that function should now be correct (more correct :)

gabyx · 2022-05-05T12:17:49Z

Related: #1737, #1944, #437

In Dockerfile, if there is something like: ``` RUN setcap cap_net_raw=+ep /path/to/binary ``` kaniko won't detect that there is a change on file `/path/to/binary` and thus discards this layer. This patch allows the hasher function to actually look at `security.capability` extended attributes.

nagpalkaran95 · 2023-12-28T06:03:41Z

This issue is still present in the latest version. Can someone else also confirm it?

gabyx · 2024-01-03T10:03:23Z

@nagpalkaran95 Do you have a minimal working example?

nagpalkaran95 · 2024-01-03T10:15:18Z

@gabyx On digging deeper, I learned the issue is not just with symlinks but entire binaries/packages. The final image does not have any packages that were installed via apt.
Similar issue link - #1547 (comment)

sample Dockerfile:

FROM ubuntu:22.04
LABEL authors="Selenium <selenium-developers@googlegroups.com>"
ARG SELENIUM_URL

#================================================
# Customize sources for apt-get
#================================================
RUN  echo "deb http://archive.ubuntu.com/ubuntu jammy main universe\n" > /etc/apt/sources.list \
  && echo "deb http://archive.ubuntu.com/ubuntu jammy-updates main universe\n" >> /etc/apt/sources.list \
  && echo "deb http://security.ubuntu.com/ubuntu jammy-security main universe\n" >> /etc/apt/sources.list

# No interactive frontend during docker build
ENV DEBIAN_FRONTEND=noninteractive \
    DEBCONF_NONINTERACTIVE_SEEN=true

#========================
# Miscellaneous packages
# Includes minimal runtime used for executing non GUI Java programs
#========================

RUN apt-get -qqy update \
  && apt-get -qqy --no-install-recommends install \
    acl \
    bzip2 \
    ca-certificates \
    openjdk-11-jre-headless \
    tzdata \
    sudo \
    unzip \
    wget \
    jq \
    curl \
    supervisor \
    gnupg2 \
  && rm -rf /var/lib/apt/lists/* /var/cache/apt/* \
  && sed -i 's/securerandom\.source=file:\/dev\/random/securerandom\.source=file:\/dev\/urandom/' /usr/lib/jvm/java-11-openjdk-amd64/conf/security/java.security

#======================================
# Add Supervisor configuration file
#======================================
COPY supervisord.conf /etc

RUN which wget
RUN which python3

CMD ["/opt/bin/entry_point.sh"]

When running the command which inside Dockerfile prints the correct path (/usr/bin/wget for example) but the path doesn't have the required binaries in the final image that is being pushed.

Also, the final image size is smaller in size than the size of the image being created using the Docker build command.

Do you have any insights for this? I have tried other workarounds suggested for enabling cache, using cleanup arg etc.

This was referenced Apr 19, 2022

Build failed with "removing whiteout sys/.wh..wh..opq: unlinkat /sys/.wh..opq: read-only file system" #1420

Closed

Fix: Flatten layer function needs to return existing files in the layer correctly #2057

Merged

imjasonh closed this as completed in #2057 Apr 26, 2022

imjasonh reopened this Apr 28, 2022

gabyx mentioned this issue Apr 28, 2022

fix: Refactor LayersMap to correct old strange code behavior #2066

Merged

4 tasks

guillaume-d mentioned this issue Apr 30, 2022

COPY to symlink destination behaviour differs from docker build #1658

Closed

imjasonh closed this as completed in #2066 May 18, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Missing symlink in the images built using Kaniko #1944

Missing symlink in the images built using Kaniko #1944

prasadkris commented Feb 18, 2022

prasus commented Mar 8, 2022

imjasonh commented Mar 8, 2022

gabyx commented Mar 17, 2022 •

edited

Loading

gabyx commented Mar 17, 2022 •

edited

Loading

gabyx commented Mar 17, 2022

gabyx commented Mar 17, 2022 •

edited

Loading

gabyx commented Mar 17, 2022 •

edited

Loading

gabyx commented Mar 18, 2022

DS-KrzJon commented Apr 6, 2022

gabyx commented Apr 28, 2022 •

edited

Loading

gabyx commented Apr 28, 2022

imjasonh commented Apr 28, 2022

gabyx commented Apr 28, 2022

gabyx commented May 5, 2022

nagpalkaran95 commented Dec 28, 2023 •

edited

Loading

gabyx commented Jan 3, 2024

nagpalkaran95 commented Jan 3, 2024

Missing symlink in the images built using Kaniko #1944

Missing symlink in the images built using Kaniko #1944

Comments

prasadkris commented Feb 18, 2022

Kaniko

Docker

prasus commented Mar 8, 2022

imjasonh commented Mar 8, 2022

gabyx commented Mar 17, 2022 • edited Loading

gabyx commented Mar 17, 2022 • edited Loading

gabyx commented Mar 17, 2022

gabyx commented Mar 17, 2022 • edited Loading

gabyx commented Mar 17, 2022 • edited Loading

gabyx commented Mar 18, 2022

DS-KrzJon commented Apr 6, 2022

gabyx commented Apr 28, 2022 • edited Loading

gabyx commented Apr 28, 2022

imjasonh commented Apr 28, 2022

gabyx commented Apr 28, 2022

gabyx commented May 5, 2022

nagpalkaran95 commented Dec 28, 2023 • edited Loading

gabyx commented Jan 3, 2024

nagpalkaran95 commented Jan 3, 2024

gabyx commented Mar 17, 2022 •

edited

Loading

gabyx commented Mar 17, 2022 •

edited

Loading

gabyx commented Mar 17, 2022 •

edited

Loading

gabyx commented Mar 17, 2022 •

edited

Loading

gabyx commented Apr 28, 2022 •

edited

Loading

nagpalkaran95 commented Dec 28, 2023 •

edited

Loading