Build fails with message No such file or directory: '/etc/mtab'

[rtarasov]

Actual behavior
I'm new to kaniko! The build fails with message "No such file or directory: '/etc/mtab'". I understand that it is white listed. Should it be? How to remove it from whitelist?

related to: https://github.com/GoogleContainerTools/kaniko/issues/345#issuecomment-420832171

Expected behavior
Build succeeds

To Reproduce
Steps to reproduce the behavior:

1. Attaching the Dockerfile

Additional Information

• Dockerfile:

FROM registry.access.redhat.com/ubi8/ubi:8.1-328

ENV SCRIPT_DIR=/usr/local/share/service
ENV BIN_DIR=/usr/local/share/bin/
ENV LIB_DIR=/usr/local/share/lib/
ENV CONFIG_DIR=/etc/service
ENV PERSIST_DATA_DIR=/var/lib/service
ENV RUNTIME_DATA_DIR=/var/run/service
ENV LICENSE_DIR=/licenses

RUN mkdir -p ${SCRIPT_DIR}/
RUN mkdir -p ${BIN_DIR}/
RUN mkdir -p ${LIB_DIR}/
RUN mkdir -p ${CONFIG_DIR}/
RUN mkdir -p ${PERSIST_DATA_DIR}/
RUN mkdir -p ${RUNTIME_DATA_DIR}/
RUN mkdir -p ${LICENSE_DIR}/

RUN groupadd -g 1001 service && \
    useradd -u 1001 -g 1001 -d ${RUNTIME_DATA_DIR} service

RUN chown -R 1001.1001 ${PERSIST_DATA_DIR}/
RUN chown -R 1001.1001 ${RUNTIME_DATA_DIR}/

VOLUME ${CONFIG_DIR}
VOLUME ${PERSIST_DATA_DIR}
VOLUME ${RUNTIME_DATA_DIR}

ADD http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-Official /etc/pki/rpm-gpg
COPY centos.repo /etc/yum.repos.d/centos.repo
 
ADD https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm epel-release-latest-8.noarch.rpm
RUN rpm -i ./epel-release-latest-8.noarch.rpm

RUN dnf install -y --nodocs dnf-plugin-ovl && dnf clean all

RUN dnf update -y

WORKDIR ${SCRIPT_DIR}/
COPY requirements.txt ${SCRIPT_DIR}/
COPY setup.sh ${SCRIPT_DIR}/

RUN /bin/bash ${SCRIPT_DIR}/setup.sh

• We use Kaniko configured within Jenkins Pod container:

spec:
  containers:
  - args:
    - "cat"
    command:
    - "/busybox/sh"
    - "-c"
    env:
    - name: "HOME"
      value: "/home/jenkins"
    image: "gcr.io/kaniko-project/executor:debug-v0.19.0"
    imagePullPolicy: "Always"
    name: "kaniko-builder"
    resources:
      limits: {}
      requests: {}
    securityContext:
      privileged: false
    tty: true
    volumeMounts:
    - mountPath: "/tmp"
      name: "workspace-volume"
      readOnly: false
    workingDir: "/tmp"

• The Jenkins console output looks like below:

INFO�[0m[0259] RUN dnf install -y --nodocs dnf-plugin-ovl && dnf clean all 
INFO�[0m[0259] cmd: /bin/sh                                 
INFO�[0m[0259] args: [-c dnf install -y --nodocs dnf-plugin-ovl && dnf clean all] 
Updating Subscription Management repositories.
Unable to read consumer identity
Subscription Manager is operating in container mode.
CentOS-8                                         10 MB/s | 6.8 MB     00:00    
Extra Packages for Enterprise Linux 8 - x86_64  6.5 MB/s | 6.4 MB     00:00    
Red Hat Enterprise Linux 8 for x86_64 - AppStre  29 MB/s |  15 MB     00:00    
Red Hat Enterprise Linux 8 for x86_64 - BaseOS   31 MB/s |  15 MB     00:00    
Red Hat Universal Base Image 8 (RPMs) - BaseOS  1.1 MB/s | 761 kB     00:00    
Red Hat Universal Base Image 8 (RPMs) - AppStre 5.8 MB/s | 3.5 MB     00:00    
Red Hat Universal Base Image 8 (RPMs) - CodeRea  15 kB/s | 9.1 kB     00:00    
Dependencies resolved.
Installing:
 dnf-plugin-ovl         noarch         0.0.3-1.el8           epel          17 k
Install  1 Package
Total download size: 17 k
Installed size: 22 k
Downloading Packages:
dnf-plugin-ovl-0.0.3-1.el8.noarch.rpm            57 kB/s |  17 kB     00:00    
Total                                            29 kB/s |  17 kB     00:00     
warning: /var/cache/dnf/epel-fafd94c310c51e1e/packages/dnf-plugin-ovl-0.0.3-1.el8.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID 2f86d6a1: NOKEY
Extra Packages for Enterprise Linux 8 - x86_64  1.6 MB/s | 1.6 kB     00:00    
Importing GPG key 0x2F86D6A1:
 Userid     : "Fedora EPEL (8) <epel@fedoraproject.org>"
 Fingerprint: 94E2 79EB 8D8F 25B2 1810 ADF1 21EA 45AB 2F86 D6A1
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-8
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                        1/1 
  Installing       : dnf-plugin-ovl-0.0.3-1.el8.noarch                      1/1 
  Running scriptlet: dnf-plugin-ovl-0.0.3-1.el8.noarch                      1/1 
  Verifying        : dnf-plugin-ovl-0.0.3-1.el8.noarch                      1/1 
Installed products updated.

Installed:
  dnf-plugin-ovl-0.0.3-1.el8.noarch                                             

Complete!
Updating Subscription Management repositories.
Unable to read consumer identity
Subscription Manager is operating in container mode.
[Errno 2] No such file or directory: '/etc/mtab'

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko	- [Yes ]
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[rtarasov]

After adding --whitelist-var-run=false getting the following error:

error building image: error building stage: failed to get filesystem from image: error removing ./var/run to make way for new symlink: unlinkat /var/run/secrets/kubernetes.io/serviceaccount/..data: read-only file system

Could you please help with this issue?

[tejal29]

@rtarasov The flag you are using removes /var/run from whitelist which is not what you want.
I am not sure if the error you are seeing is from kaniko.
Looks like its coming from RUN dnf install -y --nodocs dnf-plugin-ovl && dnf clean all unless you have deleted some log lines before the error. Does the "dnf-plugin-ovl" plugin except the /etc/mtab direcotry.
Can you check if it your base image has this dir or it gets deleted somewhere in your dockerfile?