Pulling images from a diferent private repository than context fails after v1.7.0

[atorrembo]

Actual behavior
Trying to run a pipeline in GitLab Runner using Kubernetes executor with a private repository that has a Multistage Dockerfile using a FROM image from another private repository in the same gitlab project/group (Same access credentials)

Context:
/PRIVATE_GROUP-1/PRIVATE_PROJECT1

Script in the build stage:
/kaniko/executor --context "${CI_PROJECT_DIR}" --dockerfile "${CI_PROJECT_DIR}/Dockerfile" --target "prod" --no-push

Returns:

error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "registry.gitlab.com/privategroupname/privateregistryname/image:version": creating push check transport for registry.gitlab.com failed: GET https://gitlab.com/jwt/auth?scope=PRIVATE_REGISTRY:push,pull&service=container_registry: DENIED: access forbidden

Using --no-push parameter I got another clue

INFO[0000] Retrieving image registry.gitlab.com/privateregistry/image:1.0 from registry registry.gitlab.com
error building image: GET https://gitlab.com/jwt/auth?scope=PRIVATE_REGISTRY:pull&service=container_registry: UNAUTHORIZED:
HTTP Basic: Access denied. The provided password or token is incorrect or your account has 2FA enabled and you must use a personal access token instead of a password. See https://gitlab.com/help/user/profile/account/two_factor_authentication#troubleshooting

So after trying everything I've found in the internet I started downgrading versions from v1.11.0 until in v1.7.0 the job starts to work as expected.

Expected behavior
Image built and pushed to a private repository

To Reproduce
Steps to reproduce the behavior:

1. Run a build Job in a Project having a Dockerfile using a FROM image from another project registry in the same group (same credentials). I handle the authentication by injecting the docker.json file

Additional Information

• Dockerfile
  FROM registry.gitlab.com/PRIVATE_GROUP-1/PRIVATE_REGISTRY-2/runner:3.11 AS prod
• Build Context
  Just FROM, it fails in the first step trying to pull that image
• Kaniko Image (fully qualified with digest)
  V1.7.0 OK
  V1.8.0 FAILS
  V1.9.0 FAILS
  V1.10.0 FAILS
  V1.11.0 FAILS

Part of my runners config:
# [[runners.kubernetes.volumes.secret]]
# name = "gitlab-registry-token"
# mount_path = "/root/.docker/"

And the secret:

This is only additonal info, this configuration works with "docker pull" and with kaniko < 1.7

• A "docker pull" command added for testing in the same job works OK

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing	- [No]
Please check if the build works in docker but not in kaniko	- [Yes]
Please check if this error is seen when you use --cache flag	- [No]
Please check if your dockerfile is a multistage dockerfile	- [Yes]

[lusu007]

I encountered the same problem when I set up Harbor as a proxy for DockerHub to overcome DockerHub's strict rate limiting. However, upon changing the base image to the proxy registry, I encountered this issue. 🫤

[lusu007]

Is there any update regarding this issue? DockerHub's rate limiting is causing significant disruptions to my pipeline, effectively halving its efficiency. I'm eagerly hoping for a resolution to this problem, as it would enable me to overcome the rate limit restrictions and restore my pipeline's full functionality.

[algo7]

Any news on this?

[lusu007]

Hey,
this still makes our proxy cache unusable. I would be happy to provide more information if needed and I would appreciate if someone could take a look at this.

[zijiwork]

What progress has been made?

[RufusJWB]

It would be great if you could provide a fix or a recommended work-around. This problem will probably appear more often in the next months, since Visual Studio is now creating multistage build files for container deployment of ASP.net applications: https://learn.microsoft.com/en-us/visualstudio/containers/container-build?view=vs-2022#multistage-build . I hope my reproducible problem description in #2944 will help you fixing this issue.

[atorrembo]

Have you tried using runner version 1.7.0?

[lusu007]

Have you tried using runner version 1.7.0?

Using an old outdated version isn't an optimal solution. 😅

[atorrembo]

Nope, just for investigation/debuggin purposes

[RufusJWB]

Have you tried using runner version 1.7.0?

I just did. This is the docker file I use:

FROM gcr.io/kaniko-project/executor:v1.7.0-debug

WORKDIR /buildDir/SRC

COPY . .

WORKDIR "PROXY Protocol Tester"

CMD ["--verbosity=debug", "--ignore-path=/buildDir", "--context=/buildDir", "--single-snapshot", "--cache=false", "--dockerfile=/buildDir/SRC/PROXY Protocol Tester/Dockerfile", "--no-push"]

But it is also failing: https://gist.github.com/RufusJWB/1e9b46657faea8c4ea31574fcc120c44

[DeanHnter]

Still having this exact issue with current version

[RufusJWB]

Still having this exact issue with current version

Have a look into my repro: #2944 I was able to fix it.

[lusu007]

Still having this exact issue with current version

Have a look into my repro: #2944 I was able to fix it.

I just tried your suggested fix, but it's not working for me. In my opinion #2944 and this issue aren't even related because we are having authentication issues and you are not. 🙁

Even with your fixes implemented I still get the following error:

error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "redacted.com/redacted/redacted": POST https://redacted.com/v2/redacted/redacted/blobs/uploads/: UNAUTHORIZED: unauthorized to access repository: redacted/redacted, action: push: unauthorized to access repository: redacted/redacted, action: push

[lusu007]

After further debugging, I found that the problem arises when the --no-push flag is utilized. However, when the --no-push flag isn't set, but a --destination is specified, the error disappears and authentication proceeds successfully.

I believe the no-push flag should not influence the authentication when pulling base images.

[meyfa]

With --no-push set, this code is never called:

kaniko/pkg/executor/push.go

Line 132 in d6aab15

if err := checkRemotePushPermission(destRef, creds.GetKeychain(), tr); err != nil {

Behind the scenes, that function performs authentication against the registry. As such, unless we are pushing, auth is never performed.

We would need an additional function, e.g., checkRemotePullPermissions, that runs in case --no-push is set. This would allow us to use remote images as the base image even without pushing.

[atorrembo]

damn...
setting env var DOCKER_CONFIG=/root/.docker did the trick (running on gitlab kubernetes executor]