root GitLab runner builds failing on Kaniko v1.8.0-debug and newer

[kruserr]

There seems to be a breaking change related to root GitLab runner builds in the Kaniko v1.8.0-debug release.

Actual behavior
All builds were running smoothly on v1.7.0-debug.
non-root builds are running smoothly on the newset Kaniko v1.9.1-debug.
root builds are failing on the newset Kaniko v1.9.1-debug.
With backtracing we found that the builds are failing on any Kaniko version from v.1.8.0-debug to v1.9.1-debug.

Expected behavior
root builds on GitLab runner on Kaniko versions v1.8.0-debug and newer run smoothly.

To Reproduce
.gitlab-ci.yml

...
build:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:v1.8.0-debug
    entrypoint: [""]
  script:
    - |
      if [[ -z "$CI_COMMIT_TAG" ]]; then
        export CI_APPLICATION_REPOSITORY=${CI_APPLICATION_REPOSITORY:-$CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG}
        export CI_APPLICATION_TAG=${CI_APPLICATION_TAG:-$CI_COMMIT_SHA}
      else
        export CI_APPLICATION_REPOSITORY=${CI_APPLICATION_REPOSITORY:-$CI_REGISTRY_IMAGE}
        export CI_APPLICATION_TAG=${CI_APPLICATION_TAG:-$CI_COMMIT_TAG}
      fi

      image_tagged="$CI_APPLICATION_REPOSITORY:$CI_APPLICATION_TAG"
      image_latest="$CI_APPLICATION_REPOSITORY:latest"
      
      mkdir -p /kaniko/.docker
      echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
      
      /kaniko/executor \
        --force \
        --cache=true \
        --context $CI_PROJECT_DIR \
        --dockerfile $CI_PROJECT_DIR/Dockerfile \
        $AUTO_DEVOPS_BUILD_IMAGE_EXTRA_ARGS \
        --destination "$image_tagged" \
        --destination "$image_latest"
  rules:
    - if: '$AUTO_DEVOPS_PLATFORM_TARGET == "EC2"'
      when: never
    - if: '$CI_COMMIT_TAG || $CI_COMMIT_BRANCH'
...

log:

Running with gitlab-runner 14.8.2 (foo)
  on foo-bar-gitlab-runner-foo-bar FOO
Preparing the "kubernetes" executor 00:00
Using Kubernetes namespace: foo
Using Kubernetes executor with image gcr.io/kaniko-project/executor:v1.8.0-debug ...
Using attach strategy to execute scripts...
Preparing environment 00:12
Waiting for pod foo/runner-foo-project-17-concurrent-foo to be running, status is Pending
Waiting for pod foo/runner-foo-project-17-concurrent-foo to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod foo/runner-foo-project-17-concurrent-foo to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod foo/runner-foo-project-17-concurrent-foo to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Running on runner-foo-project-17-concurrent-foo via foo-bar-gitlab-runner-foo-bar...
Getting source from Git repository 00:01
Fetching changes with git depth set to 50...
Initialized empty Git repository in /foo/bar/.git/
Created fresh repository.
Checking out foo as master...
Skipping Git submodules setup
Executing "step_script" stage of the job script 00:01
$ if [[ -z "$CI_COMMIT_TAG" ]]; then # collapsed multi-line command
WARN[0000] kaniko is being run outside of a container. This can have dangerous effects on your system 
INFO[0000] Retrieving image manifest registry.foo.bar.foo/bar/foo 
INFO[0000] Retrieving image registry.foo.bar.foo/bar/foo from registry registry.foo.bar.foo 
INFO[0000] Retrieving image manifest registry.foo.bar.foo/bar/foo 
INFO[0000] Returning cached image manifest              
INFO[0000] Built cross stage deps: map[]                
INFO[0000] Retrieving image manifest registry.foo.bar.foo/bar/foo 
INFO[0000] Returning cached image manifest              
INFO[0000] Retrieving image manifest registry.foo.bar.foo/bar/foo 
INFO[0000] Returning cached image manifest              
INFO[0000] Executing 0 build triggers                   
INFO[0000] Checking for cached layer registry.foo.bar.foo/foo/bar/master/cache:foo... 
INFO[0000] No cached layer found for cmd RUN apt-get update && apt-get install -y libvips && rm -rf /var/lib/apt/lists/* 
INFO[0000] Unpacking rootfs as cmd RUN apt-get update && apt-get install -y libvips && rm -rf /var/lib/apt/lists/* requires it. 
error building image: error building stage: failed to get filesystem from image: failed to write "security.capability" attribute to "/usr/bin/newgidmap": operation not permitted
Cleaning up project directory and file based variables 00:00
ERROR: Job failed: command terminated with exit code 1

Additional Information

• Dockerfile
  • Any docker file that uses root priviliges
• Kaniko Image (fully qualified with digest)
  • gcr.io/kaniko-project/executor:v1.8.0-debug sha256:0d8408715c7bcc2dc747405936c0e72665cafb2357fb78e23eb71f90bc39624f
  • gcr.io/kaniko-project/executor:v1.9.1-debug sha256:ac169723b2076f9d5804f4bc05c98397e286da6fdcdd5a09fdc179f06ccb3be1

Potentially related to these:

• Fix copying ownership #1725
• Fix calculating path for copying ownership #1859

Related issues:

• build fails with filesystem permission error: failed to write "security.capability" attribute #2201

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[aaron-prindle]

@kruserr can you confirm if this issue is still occurring w/ the latest version of Kaniko - v1.12.1?

[CaptainGlac1er]

@aaron-prindle it is still happening for me with v1.12.1

[CaptainGlac1er]

@aaron-prindle same is happening with v1.14.0

[CaptainGlac1er]

@aaron-prindle still is happening with v1.16.0

[CaptainGlac1er]

@aaron-prindle is it safe to use the ignore dir to work around this issue?
#2201

[CaptainGlac1er]

still is happening with v1.17.0

[CaptainGlac1er]

Still happening with v1.20.0-debug

[CaptainGlac1er]

I wonder if it is related to this.
e65bce1#diff-a7dc2085daa1b9679520b0f79b134a5215227e6e20424ed95ff338a02733ddf1R166