Kaniko fails to push built image to a private registry with self-signed certificate

[netcho]

Actual behavior
When building an image with skaffold, kaniko fails to push the image after it has built it. The build process goes fine but when the image has to be pushed to the registry, I get the following error:

failed to build: getting image: Get "https://registry.home/v2/": x509: certificate signed by unknown authority

I have specified a certificate file with --registry-certificate flag via skaffold.yaml. The registry uses a certificate chain with the following order:

Root CA -> Intermediate CA -> Server cert.

I have also tried mounting the certificate chain as /kaniko/ssl/certs/additional-ca-cert-bundle.crt but kaniko fails with the same error.

Expected behavior
Kaniko should successfully push the image to registry.

To Reproduce
Steps to reproduce the behavior:

1. Run Harbor with self-signed certificate chain
2. Run a build with skaffold using kaniko as in-cluster build tool

Additional Information

• Dockerfile
  FROM node:10 AS build-env
  ADD . /app
  WORKDIR /app

  RUN npm install --only=production

  FROM gcr.io/distroless/nodejs:10
  COPY --from=build-env /app /app
  WORKDIR /app
  CMD ["server.js"]

• A sample nodejs app

• Logs from the build: https://paste2.org/bLvmzUEN

• Kaniko Image: latest debug

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[kennethredler]

It may be worth double-checking the /kaniko/ssl/certs/ca-certificates.crt contents to be certain they are well formed and contain all of the certificates in the chain.

This has worked well for me in various contexts:

cat "$REGISTRY_CERT" >> /kaniko/ssl/certs/ca-certificates.crt   # File at path in $REGISTRY_CERT contains all certs in chain

[netcho]

I have checked it several times. /kaniko/ssl/certs/ca-certificates.crt contains the preconfigured Root CAs necessary for communicating with gcr, docker hub etc. In skaffold one can't issue a command prior to calling kaniko executor (or it's just me). The solution I found that kind of fools kaniko is to mount a secret as /kaniko/ssl/certs/additional-ca-cert-bundle.crt. This I managed to get working so that the build starts but when trying to push the image, kaniko fails. I get the same result if I mount the secret in the pod (doesn't matter where exactly) and pass --registry-certificate argument to kaniko.

[vistaarjuneja]

@netcho were you able to get this to work? I have tried adding the cert in /kaniko/ssl/certs/ca-certificates.crt but seem to be getting the same error. Any help would be greatly appreciated!

[faenigma]

Anyone have updates on this?

I'm running into cert issues with my pipeline that weren't happening previously (numerous successful builds over the past 6 months) and I'm kind of lost as to what my next steps should be. Nothing with my certs had changed, though I've made sure to insert certs and verify that they are in for both those in /kaniko/ssl/certs/ca-certificates.crt and by creating and adding them to /kaniko/ssl/certs/additional-ca-cert-bundle.crt as described above.

Rocky Linux 8 - AppStream                       0.0  B/s |   0  B     00:02    
Errors during downloading metadata for repository 'appstream':
  - Curl error (60): Peer certificate cannot be authenticated with given CA certificates for https://mirrors.rockylinux.org/mirrorlist?arch=x86_64&repo=AppStream-8 [SSL certificate problem: self signed certificate in certificate chain]
Error: Failed to download metadata for repo 'appstream': Cannot prepare internal mirrorlist: Curl error (60): Peer certificate cannot be authenticated with given CA certificates for https://mirrors.rockylinux.org/mirrorlist?arch=x86_64&repo=AppStream-8 [SSL certificate problem: self signed certificate in certificate chain] 

[aaronsuns]

Also got the same problem, any work around? Tried to put the certifcate after /kaniko/ssl/certs/ca-certificates.crt, but not luck.

[tooptoop4]

same issue, i wonder if old certs from the build time of kaniko itself are the issue

[igorgbianchi]

Same problem here, we were using Kaniko in the last two years to build Docker images inside Gitlab runner jobs without any problems, but this week we have to update our Docker registry cert and this error started to happen. I can push to our private registry from k8s nodes and from my PC, but not from a pipeline job running Kaniko.

It looks similar to #2281, however, the 1.9.2 version didn't fix it.

[markusmauch]

I have the same issue. This is not fixed, presumably?

[taislapta]

as workaround you can copy your private registry cert in to kaniko executor image /kaniko/ssl/certs/ and it will work

[dfarley1]

We're running into the same issue, unfortunately @taislapta's suggestion didn't work either. We've tried:

• Appending to ca-certificates.crt:

FROM gcr.io/kaniko-project/executor:debug-v0.14.0
COPY certs/custom_cert.crt /kaniko/ssl/certs/
RUN cat /kaniko/ssl/certs/custom_cert.crt >> /kaniko/ssl/certs/ca-certificates.crt


INFO[0000] Building stage 'gcr.io/kaniko-project/executor:v1.14.0-debug' [idx: '0', base-idx: '-1'] 
INFO[0000] Unpacking rootfs as cmd COPY certs/custom_cert.crt /kaniko/ssl/certs/ requires it. 
INFO[0004] Initializing snapshotter ...                 
INFO[0004] Taking snapshot of full filesystem...        
INFO[0005] COPY certs/custom_cert.crt /kaniko/ssl/certs/ 
INFO[0005] RUN cat /kaniko/ssl/certs/custom_cert.crt >> /kaniko/ssl/certs/ca-certificates.crt 
INFO[0005] Cmd: /bin/sh                                 
INFO[0005] Args: [-c cat /kaniko/ssl/certs/custom_cert.crt >> /kaniko/ssl/certs/ca-certificates.crt] 
INFO[0005] Running: [/bin/sh -c cat /kaniko/ssl/certs/custom_cert.crt >> /kaniko/ssl/certs/ca-certificates.crt] 
INFO[0005] Taking snapshot of full filesystem...        
INFO[0005] No files were changed, appending empty layer to config. No layer added to image. 
INFO[0005] Deleting filesystem...                       
INFO[0005] Pushing image to ...

• Directly overwriting ca-certificates.crt:

FROM gcr.io/kaniko-project/executor:v1.14.0-debug
COPY certs/custom_cert.crt /kaniko/ssl/certs/
COPY certs/custom_cert.crt /kaniko/ssl/certs/ca-certificates.crt


INFO[0000] Building stage 'gcr.io/kaniko-project/executor:v1.14.0-debug' [idx: '0', base-idx: '-1'] 
INFO[0000] Unpacking rootfs as cmd COPY certs/custom_cert.crt /kaniko/ssl/certs/ca-certificates.crt requires it. 
INFO[0002] Initializing snapshotter ...                 
INFO[0002] Taking snapshot of full filesystem...        
INFO[0002] COPY certs/custom_cert.crt /kaniko/ssl/certs/ca-certificates.crt 
INFO[0002] Taking snapshot of full filesystem...        
INFO[0002] No files were changed, appending empty layer to config. No layer added to image. 
INFO[0002] Deleting filesystem...                       
INFO[0002] Pushing image to ...

Both of which should result in our custom cert living in /kaniko/ssl/certs as well as ca-certificates.crt, but in both cases we see "No files were changed" and we are unable to use the resulting image to push images to our private registry.

We are able to append our cert to ca-certificates.crt at "runtime" by doing the following in our image building jobs:

before_script:
    - cat gitlab/custom_cert.crt >> /kaniko/ssl/certs/ca-certificates.crt
script:
    - /kaniko/executor --dockerfile "${DOCKERFILE}" --destination "${DESTINATION}" --context $(pwd)/build

But this requires us to store the cert in every repo we want to build images for.

[jimsnab]

But this requires us to store the cert in every repo we want to build images for.

We are using a similar approach, but the CA cert comes from the build environment similar to your ${DESTINATION}, e.g.,

echo ${ARTIFACTORY_CA} >> /kaniko/ssl/certs/ca-certificates.crt

We build with Jenkins, and the apps have a Jenkins pipeline that reference our CA environment variable but do not need to store a cert in the app repo. It has been working well.

We also derived a custom Kaniko container that had the cert pre-installed and that worked too, but opted for the solution above to avoid maintaining image updates.

[dfarley1]

We also derived a custom Kaniko container that had the cert pre-installed and that worked too, but opted for the solution above to avoid maintaining image updates.

If you notice the Dockerfile script in my first code block, that's exactly what we're trying to do! But the kaniko image that kaniko builds doesn't end up with the cert installed.

We found a root cause in our case being the --single-snapshot flag. Removing that allows us to create a custom Kaniko container with the custom cert installed with a regular old COPY. Supposedly the only downside is a slightly larger image, but I can't confirm or deny any other side effects.

[sebglon]

I have this issue when i try a build a kaniko extended image with the arg --cleanup
if i remove the push image work well