Latest release regresses OCI layout support

[mattmoor]

Actual behavior

With the latest release (last night), some of our e2e tests fail with:

INFO[0000] CMD ["echo Hello World"]                     
INFO[0001] Skipping push to container registry due to --no-push flag 

Error: open /var/run/oci/layout/blobs/sha256/c6f4d1a13b699c8490910fd4fd6c7056b90fd0da3077e4f29b4bd27bf0bae6cd: permission denied

Expected behavior

This passed with :latest prior to the v1.8.0 cut, which according to @imjasonh was v1.6.0.

To Reproduce

These are the two relevant steps of the Tekton task whose output you see above:

    - name: build-as-layout
      image: gcr.io/kaniko-project/executor:latest
      args:
      - --dockerfile=/workspace/$(params.path)/$(params.dockerfile)
      - --context=/workspace
      - --no-push # We are writing a layout
      - --oci-layout-path=/var/run/oci/layout
      - --digest-file=/tekton/results/dev.mink.images.digest
      - $(params.kaniko-args)
      volumeMounts:
      - name: oci-layout
        mountPath: /var/run/oci/layout

    - name: push-layout
      image: gcr.io/go-containerregistry/krane:latest
      args: ["push", "--image-refs=/tekton/results/IMAGES", "/var/run/oci/layout", "$(params['dev.mink.images.target'])"]
      volumeMounts:
      - name: oci-layout
        mountPath: /var/run/oci/layout

Here krane used to be able to read the layout and publish it (it is :nonroot), but something about the latest release is causing the permission denied error above.

Pinning to v1.7.0 "fixed" this for us, which narrows the window of regression to sometime in v1.8.0.

Additional Information

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[mattmoor]

cc @priyawadhwa @tejal29

[imjasonh]

Possibly related changes since 1.7.0:

• Fix calculating path for copying ownership #1859
• Fix possible nil pointer derefence in fs_util.go #1813
• Fixes #1837 : keep file capabilities on archival #1838 (doubt it but you never know)

[thomas-tacquet]

By switching from 1.7.0 to 1.8.0 I got the same behaviour but I'm not sure it has something to do with OCI, it's more general (worse)

I just push my built image on my destination (private container registry) and it keeps failing. Reverted to 1.7.0 and everything is fine.

if it can help troublehsooting, here are my kaniko args :

        - "--use-new-run"
        - "--cache=true" 
        - "--cache-dir=/XXX" 
        - "--snapshotMode=redo"
        - "--dockerfile=MYDOCKERFILE"
        - "--destination=MYREGISTRY"

error : error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "PRIVATE_REGISTRY"

[robertlestak]

Confirming we saw the same (error checking push permissions) with :latest, pinning to :1.7.0 "fixed" it.

[imjasonh]

"error checking push permissions" issue are not related to the originally reported bug (which doesn't involve pushing at all, FWIW).

[BernhardGruen]

Did someone test if this regression is solved with v1.8.1?

[thomas-tacquet]

Tested in v1.8.1 and I still have the same problems, investigating.