fix rootfs after execute '/kaniko/executor xxx --cleanup'

[hq-git]

Actual behavior

I have build an custom image since I need git and other commands during build stage, I found that after execute '/kaniko/executor xxx --cleanup' to build multi stages image, some folders of rootfs disappeared.

Expected behavior

Is it doable to have a option like '--backup' to backup rootfs before executing the '/kaniko/executor xxx --cleanup', and restore the rootfs when build is finished?

To Reproduce

In my custom kaniko executor image(base on debian 11 and kaniko v1.15.0).

## Kaniko basic image
FROM gcr.io/kaniko-project/executor:v1.15.0-debug AS kaniko

## Final basic image
FROM maven:3.8.6-openjdk-8-slim

# Install necessary tools
USER root
COPY ./debian11-sources.list /etc/apt/sources.list
RUN set -eux; \
    apt-get update; \
    apt-get install -y --no-install-recommends ca-certificates curl git git-lfs; \
    apt-get install -y --no-install-recommends openssh-client patch; \
    apt-get clean; \
    rm -rf /var/lib/apt/lists/*

# Install kaniko. Create kaniko directory with world write permission to allow non root run
COPY --from=kaniko /kaniko /kaniko
COPY --from=kaniko /busybox /kaniko_bin
RUN chmod 777 /kaniko

# Declare '/kaniko_bin' as a volume to get it automatically in the path which kaniko to ignore
VOLUME /kaniko_bin
# Move necessary commands to '/kaniko' directory, in case be cleaned by kaniko.
RUN set -eux; \
    mv $(which git) /kaniko_bin/; \
    mv $(which git-lfs) /kaniko_bin/; \
    usermod root -s /kaniko_bin/sh; \
    mkdir -p ${HOME}/.ssh && chmod 700 ${HOME}/.ssh

# Set kaniko env
ENV HOME="/root"
ENV USER="root"
ENV PATH="/kaniko:/kaniko_bin:$PATH"
ENV SSL_CERT_DIR="/kaniko/ssl/certs"
ENV DOCKER_CONFIG="/kaniko/.docker/"
ENV DOCKER_CREDENTIAL_GCR_CONFIG="/kaniko/.config/gcloud/docker_credential_gcr_config.json"

WORKDIR /workspace

ENTRYPOINT [ "/kaniko/executor" ]
CMD [ "--help" ]

I found that after execute '/kaniko/executor xxx --cleanup' to build multi stages image, some folders of rootfs disappeared.

before:

/workspace # ls -l /
total 92
drwxr-xr-x    1 root     root          4096 Aug  3  2022 bin
drwxr-xr-x    2 root     root          4096 Jun 30  2022 boot
drwxr-xr-x    5 root     root           360 Sep 27 08:10 dev
drwxr-xr-x    1 root     root          4096 Sep 27 08:10 etc
drwxr-xr-x    2 root     root          4096 Jun 30  2022 home
drwxrwxrwx    1 root     root          4096 Aug 29 21:49 kaniko
drwxr-xr-x    2 root     root         12288 Sep 27 08:10 kaniko_bin
drwxr-xr-x    1 root     root          4096 Aug  1  2022 lib
drwxr-xr-x    2 root     root          4096 Aug  1  2022 lib64
drwxr-xr-x    2 root     root          4096 Aug  1  2022 media
drwxr-xr-x    2 root     root          4096 Aug  1  2022 mnt
drwxr-xr-x    2 root     root          4096 Aug  1  2022 opt
dr-xr-xr-x  235 root     root             0 Sep 27 08:10 proc
drwx------    1 root     root          4096 Sep 27 08:10 root
drwxr-xr-x    3 root     root          4096 Aug  1  2022 run
drwxr-xr-x    1 root     root          4096 Aug  3  2022 sbin
drwxr-xr-x    2 root     root          4096 Aug  1  2022 srv
dr-xr-xr-x   13 root     root             0 Sep 27 08:10 sys
drwxrwxrwt    1 root     root          4096 Sep 27 07:47 tmp
drwxr-xr-x    1 root     root          4096 Aug  1  2022 usr
drwxr-xr-x    1 root     root          4096 Aug  1  2022 var
drwxr-xr-x    7 root     root           224 Sep 27 07:47 workspace

after:

/workspace # ls -l /
total 44
drwxr-xr-x    5 0        0              360 Sep 27 07:49 dev
drwxr-xr-x    1 0        0             4096 Sep 27 07:57 etc
drwxrwxrwx    1 0        0             4096 Sep 27 07:57 kaniko
drwxr-xr-x    2 0        0            12288 Sep 27 07:49 kaniko_bin
dr-xr-xr-x  229 0        0                0 Sep 27 07:49 proc
dr-xr-xr-x   13 0        0                0 Sep 27 07:49 sys
drwxrwxrwt    1 0        0             4096 Sep 27 07:57 tmp
drwxr-xr-x    1 0        0             4096 Sep 27 07:57 var
drwxr-xr-x   17 0        0              544 Sep 26 10:57 workspace

Even I moved the 'git' command to '/kaniko_bin/', I still got below errors, but the 'patch' command still works. I think it's because 'patch' command is an binary command from kaniko image, it has no external dependency. But for 'git' command, it relies on some dependencies under rootfs.

/workspace # which git
/kaniko_bin/git
/workspace # ls -l /kaniko_bin/git
-rwxr-xr-x    1 0        0          3347408 Feb 22  2023 /kaniko_bin/git
/workspace # git -h
/kaniko_bin/sh: git: not found

/workspace # which patch
/kaniko_bin/patch
/workspace # ls -l /kaniko_bin/patch
-rwxr-xr-x  398 0        0          1161032 Jul 17 19:27 /kaniko_bin/patch
/workspace # patch --help
BusyBox v1.36.1 (2023-07-17 19:24:58 UTC) multi-call binary.

Usage: patch [-RNE] [-p N] [-i DIFF] [ORIGFILE [PATCHFILE]]

	-p N	Strip N leading components from file names
	-i DIFF	Read DIFF instead of stdin
	-R	Reverse patch
	-N	Ignore already applied patches
	-E	Remove output files if they become empty
	--dry-run	Don't actually change files

Since the '/etc/passwd' is also deleted by kaniko, I can not enter the kaniko container, so I can not do next build with this kaniko container.

$ docker exec -ti recursing_turing /kaniko_bin/sh
unable to find user root: no matching entries in passwd file

---

I know kaniko will extract the files of base image in Dockerfile to current container's rootfs, if I used ADD or COPY in Dockerfile.

Unpacking rootfs as cmd COPY --from=builder /app/ ./ requires it.

Is it doable to have a option like '--backup' to backup rootfs before executing the '/kaniko/executor xxx --cleanup', and restore the rootfs when build is finished?

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing	- [Yes ]
Please check if the build works in docker but not in kaniko	- [No ]
Please check if this error is seen when you use --cache flag	- [ No]
Please check if your dockerfile is a multistage dockerfile	- [Yes ]

[hq-git]

I have added a script to my custom kaniko image, using it to backup rootfs before executing the '/kaniko/executor xxx --cleanup', and restore the rootfs when build is finished.

kaniko_executor.sh

#!/kaniko_bin/sh

set -e

BACKUP_DIR="/kaniko/backup"
# Directories need to backup, the most important ones are "/etc /usr /lib /lib64"
TARGET_DIR="/etc /usr /lib /lib64 /root /sbin /bin /var"

# Backup the important directories under rootfs.
rootfs_backup () {
  echo "-> Backup important directories under rootfs ..."
  if [ ! -d "${BACKUP_DIR}" ]; then mkdir -p "${BACKUP_DIR}"; fi
  for dir_name in ${TARGET_DIR}; do
      if [ -d "${BACKUP_DIR}/${dir_name}" ]; then continue; fi
      cp -aR "${dir_name}" "${BACKUP_DIR}/"
  done
}

# Restore the directories
rootfs_restore () {
  echo "-> Restore important directories under rootfs ..."
  set +e
  for dir_name in $(ls "${BACKUP_DIR}" |xargs); do
      rm -rf "/${dir_name}" 2> /dev/null
      cp -aR "${BACKUP_DIR}/${dir_name}" / 2> /dev/null
  done
  echo "-> Finished to restore rootfs, ready for the next build !"
}

# Execute the kaniko executor command
run_executor () {
  rootfs_backup
  /kaniko/kaniko_executor $*
  rootfs_restore
}

# Main
case "$1" in
    restore)
      rootfs_restore ;;
    *)
      run_executor $* ;;
esac

And added following codes to my Dockerfile

# Add backup and restore rootfs process.
COPY ./kaniko_executor.sh /kaniko/kaniko_executor.sh
RUN set -eux; \
    chmod +x /kaniko/kaniko_executor.sh; \
    mv /kaniko/executor /kaniko/kaniko_executor; \
    mv kaniko/kaniko_executor.sh /kaniko/executor

After added above to my custom kaniko image, although the git command is not working during executing the '/kaniko/executor xxx --cleanup' command, but it will work after command is finished (the shell script will restore the rootfs of container).