registry-creds addon: secrets stored with different name to defaults

[ohookins]

Environment:

Minikube version (use minikube version): v0.26.1

• OS (e.g. from /etc/os-release): MacOS 10.13.4 High Sierra
• VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName): hyperkit
• ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep -i ISO or minikube ssh cat /etc/VERSION): v0.26.0
• Install tools: homebrew

What happened:
Using private ECR registry images with registry-creds addon.

What you expected to happen:
Installing and configuring registry-creds with valid credentials would allow ECR images to be retrieved by minikube while creating pods.

How to reproduce it (as minimally and precisely as possible):

1. minikube addons configure registry-creds
2. Configure the credentials with a valid private AWS keypair and ECR registry ID/region.
3. Configure a deployment/replicaset/replicationcontroller/etc with an image located in that private registry.
4. Image fails to download.

Output of minikube logs (if applicable):

2018-05-11 16:53:17 +0200 CEST   2018-05-11 16:51:41 +0200 CEST   4         user-interface-9789bc6d8-wwjns.152d9ea9f11e04d0    Pod          spec.containers{user-interface}   Warning   Failed                    kubelet, minikube       Failed to pull image "XXXX.dkr.ecr.us-east-1.amazonaws.com/XXXX/XXXX": rpc error: code = Unknown desc = Error response from daemon: Get https://XXXX.dkr.ecr.us-east-1.amazonaws.com/v2/XXXX/XXXX/manifests/latest: no basic auth credentials

Anything else do we need to know:

• Installing the registry-creds addon causes credentials to be stored in secrets named as follows:

$ k get secrets --namespace=kube-system | grep registry-creds                                               17:27:36
registry-creds-dpr                               Opaque                                3         35m
registry-creds-ecr                               Opaque                                6         35m
registry-creds-gcr                               Opaque                                2         35m

• According to the source of registry-creds, the default secret name for AWS credentials in fact awsecr-cred
• No configuration seems to be provided to registry-creds to override this value, so it is left looking for secrets that don't exist:

time="2018-05-11T15:08:27Z" level=error msg="Error getting secret: secrets "awsecr-cred" not found"
2018/05/11 15:08:27 Finished processing secret for namespace default, secret awsecr-cred
time="2018-05-11T15:08:27Z" level=error msg="Error getting secret: secrets "dpr-secret" not found"
2018/05/11 15:08:27 Finished processing secret for namespace default, secret dpr-secret
2018/05/11 15:08:27 Refreshing credentials...
time="2018-05-11T15:08:27Z" level=info msg="------------------ [gcr-secret] ----------------------
"
time="2018-05-11T15:08:27Z" level=info msg="Error getting secret for provider gcr-secret. Skipping secret provider! [Err: google: error getting credentials using well-known file (/root/.config/gcloud/application_default_credentials.json): invalid character 'c' looking for beginning of value]"

• Re-storing the secret at the default path appears to work.

[aaron-prindle]

The flow for registry-creds is:
minikube addons configure registry-creds creates the initial secret
minikube addons enable registry-creds runs the registry creds addon
can you verify that you also enabled the addon with minikube addons enable registry creds

[ohookins]

Yes, I ran both of these steps.

[adampl]

There is a similar issue with Docker Registry secret on the (currently latest) version 0.28.

Minikube logs:

Unable to retrieve pull secret default/docker-hub for default/application-5cd5488b7d-kqh8r due to secrets "docker-hub" not found.  The image pull may not succeed.

It's looking for a docker-hub secret while it is named dpr-secret.

As a workaround, I created a copy of the dpr-secret renamed to docker-hub.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[tstromberg]

This sounds like an annoying behavior. Help wanted!

[Conky5]

To re-copy the secrets with the correct names:

kubectl --namespace=kube-system get secret registry-creds-dpr -o yaml \
  | sed 's/registry-creds-dpr/dpr-secret/' \
  | kubectl create -f -
kubectl --namespace=kube-system get secret registry-creds-ecr -o yaml \
  | sed 's/registry-creds-ecr/awsecr-cred/' \
  | kubectl create -f -
kubectl --namespace=kube-system get secret registry-creds-gcr -o yaml \
  | sed 's/registry-creds-gcr/gcr-secret/' \
  | kubectl create -f -

[jmcclell]

Ran into this today. Some more info for anyone who may feel like tackling this (I might give it a go if I can find some time):

If you look at the generated replication controller manifest for the registry-cred addon, it's clear that the goal is to be able to eschew the registry-cred args all-together and override them via env vars, a la:

spec:                                                                                                                                                                                                                                                                         
  containers:                                                                                                                                                                                                                                                                 
  - env:                                                                                                                                                                                                                                                                      
    - name: AWS_ACCESS_KEY_ID                                                                                                                                                                                                                                                 
      valueFrom:                                                                                                                                                                                                                                                              
        secretKeyRef:                                                                                                                                                                                                                                                         
          key: AWS_ACCESS_KEY_ID                                                                                                                                                                                                                                              
          name: registry-creds-ecr                                                                                                                                                                                                                                            
    - name: AWS_SECRET_ACCESS_KEY                                                                                                                                                                                                                                             
     valueFrom:                                                                                                                                                                                                                                                              
        secretKeyRef:                                                                                                                                                                                                                                                         
          key: AWS_SECRET_ACCESS_KEY                                                                                                                                                                                                                                          
          name: registry-creds-ecr                                                                                                                                                                                                                                            
    - name: AWS_SESSION_TOKEN
# ...snip

Luckily, registry-creds allows for this!

https://github.com/upmc-enterprises/registry-creds/blob/master/main.go#L350

// ...snip
func validateParams() {
	// Allow environment variables to overwrite args
	awsAccountIDEnv := os.Getenv("awsaccount")
	awsRegionEnv := os.Getenv("awsregion")
	argAWSAssumeRoleEnv := os.Getenv("aws_assume_role")
	dprPassword := os.Getenv(dockerPrivateRegistryPasswordKey)
	dprServer := os.Getenv(dockerPrivateRegistryServerKey)
	dprUser := os.Getenv(dockerPrivateRegistryUserKey)
	gcrURLEnv := os.Getenv("gcrurl")
/// ...snip

....for some parameters. Unfortunately, the AWS access credentials are not included for some odd reason.

It looks like there are 3 options to fix this:

1. The code that generates the configuration can revert to using the configmap names expected by default by registry-cred
2. The code that generates the replication controller can pass in the config map names as args to the container
3. registry-cred itself can be updated to allow ENV vars to override the remaining parameters

[tstromberg]

@jmcclell - option #1 sounds good. help wanted!

[bricelalu]

Hello, ok @tstromberg, I will try to propose something very soon.

[bricelalu]

/assign

[leynebe]

To re-copy the secrets with the correct names:

kubectl --namespace=kube-system get secret registry-creds-dpr -o yaml \
  | sed 's/registry-creds-dpr/dpr-secret/' \
  | kubectl create -f -
kubectl --namespace=kube-system get secret registry-creds-ecr -o yaml \
  | sed 's/registry-creds-ecr/awsecr-cred/' \
  | kubectl create -f -
kubectl --namespace=kube-system get secret registry-creds-gcr -o yaml \
  | sed 's/registry-creds-gcr/gcr-secret/' \
  | kubectl create -f -

I ran into this ImagePullBackOff issue when deploying an application on minikube with an image stored in AWS ECR and the code in this comment above resolves it.

[ben@work .helm]$ minikube version
minikube version: v1.19.0
commit: 15cede53bdc5fe242228853e737333b09d4336b5-dirty
[ben@work .helm]$ minikube addons list
...
| default-storageclass        | minikube | enabled ✅   |
...
| registry-creds              | minikube | enabled ✅   |
| storage-provisioner         | minikube | enabled ✅   |
...
[ben@work .helm]$ cat ~/.minikube/machines/minikube/config.json | grep DriverName
    "DriverName": "docker",
[ben@work .helm]$ cat /etc/os-release 
NAME="Arch Linux"

Followed this doc https://minikube.sigs.k8s.io/docs/tutorials/configuring_creds_for_aws_ecr/ completely, troubleshooing section did not help either until I stmbled upon this issue and executed the above code did the image get successfully pulled.
It would also be greatly appreciated if the docs could list an IAM policy with minimal needed permissions for this to work.

[dobesv]

I set this up today and did not run into this issue. Maybe it is fixed now?

[sanilpaul]

Ran into this issue with minikube version 1.33.1 on mac