skaffold kaniko permission failure on getting image last step after build

[valdestron]

related to: #1719
I am facing similar issue.

Expected behavior

Do not fail

Actual behavior

After build fails to get image.

yarn run v1.22.0
$ tsc --build tsconfig.json
Done in 4.33s.
INFO[0036] Taking snapshot of full filesystem...        
INFO[0038] USER node                                    
INFO[0038] cmd: USER                                    
INFO[0038] EXPOSE 3001                                  
INFO[0038] cmd: EXPOSE                                  
INFO[0038] Adding exposed port: 3001/tcp                
INFO[0038] CMD ["yarn", "prod:run"]                     
time="2020-03-25T06:25:34Z" level=fatal msg="build failed: building [xxx.dkr.ecr.eu-central-1.amazonaws.com/xx/xx/api-gateway]: getting image: GET https://xxx.dkr.ecr.eu-central-1.amazonaws.com/v2/xx/xx/api-gateway/manifests/0.0.0-SNAPSHOT-PR-10-10: unsupported status code 401; body: Not Authorized\n"

Information

Skaffold version: v1.6
Operating system: unix
Contents of skaffold.yaml:

apiVersion: skaffold/v2beta1
kind: Config
build:
  artifacts:
  - image: ...imagename
    context: .
    kaniko:
      env: []
  cluster:
    pullSecretName: awsecr-cred
    pullSecretMountPath: /root/.aws/
    dockerConfig:
      secretName: jenkins-docker-cfg
    namespace: jx

getting image error, which happens in remoteImage() function in docker library.

awsecr-cred

kind: Secret
apiVersion: v1
metadata:
  name: awsecr-cred
  namespace: jx
  selfLink: "/api/v1/namespaces/jx/secrets/awsecr-cred"
  uid: c99686e1-6e0e-11ea-9801-022a44e5ed88
  resourceVersion: '951361'
  creationTimestamp: '2020-03-24T20:33:57Z'
data:
  ".dockerconfigjson": <dockerconfig json contents>
type: kubernetes.io/dockerconfigjson

awsecr-cred dockerconfig auth token is token got from aws cli v2 $(aws ecr get-authorization-token)

Maybe the construction of secret awsecr-cred is wrong ?

[valdestron]

Same issue as this one

[valdestron]

Ok issue solved.

I was running skaffold inside builder in the CICD Pipeline.

Kaniko is working correctly, using provided dockerconfig.json for image push.

What was not working that the builder was not using credentials. This particular case applies to jenkins-x.

pipelineConfig:
  pipelines:
    pullRequest:
      pipeline:
        options:
          distributeParallelAcrossNodes: true
        agent:
          image: gcr.io/jenkinsxio/builder-nodejs12x:latest
        stages:
          - name: build-and-push
            options: # I forgot to mount ecr creds helper on builder also
                volumes:
                  - name: docker-config
                    secret:
                      secretName: jenkins-docker-cfg
                containerOptions:
                  volumeMounts:
                    - name: docker-config
                      mountPath: /builder/home/.docker/
            parallel:
              - name: batch-1-gateways
                steps:
                  - name: version
                    sh: export VERSION=$PREVIEW_VERSION
                    # sh: yarn install
                    sh: echo 0
                  - name: npm-test
                    # sh: CI=true DISPLAY=:99 yarn test
                    sh: echo 0
                  - name: conditionally-build-api-gateway-service
                    sh: ./build.sh api-gateway
                  - name: conditionally-build-auth-service
                    sh: ./build.sh auth-service

Before I mounted the docker config it looked like this:

{"auths":{}}

After I mounted the credentials helper config to the builder image it looked like this:

{
  "credHelpers": {
    "xxxx.dkr.ecr.xxxx.amazonaws.com": "ecr-login"
  }
}

And the build completed like this:

yarn run v1.22.0
$ tsc --build tsconfig.json
Done in 4.40s.
INFO[0037] Taking snapshot of full filesystem...        
INFO[0039] USER node                                    
INFO[0039] cmd: USER                                    
INFO[0039] EXPOSE 3001                                  
INFO[0039] cmd: EXPOSE                                  
INFO[0039] Adding exposed port: 3001/tcp                
INFO[0039] CMD ["yarn", "prod:run"]                     
time="2020-03-25T16:45:39Z" level=warning msg="error adding artifacts to cache; caching may not work as expected: inspecting image: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"

Everything works as expected.