imagePullSecrets debugging help

[lgebhardt]

I'm just getting started with flux and helm. I'm trying to setup a GitOps project like the one described in https://github.com/fluxcd/helm-operator-get-started

My image is built on docker hub and I have created my credential secret like

kubectl create secret docker-registry regcred \
--docker-server="https://index.docker.io/v1/" \
--docker-username="myusername" \
--docker-password="REDACTED" \
--docker-email="larry@REDACTED" \
-n fluxcd

I'm using v1.16.2 of kubectl, and I've seen #1596. I verified the secrets contain an auth section and they work fine when using helm directly. I have also tried placing the secrets in all the namespaces as a test measure.

My setup pulls from github and reads the HelmReleases fine. However in the logs I get:

ts=2019-11-07T21:34:27.637321039Z caller=warming.go:180 component=warmer canonical_name=index.docker.io/myname/myimage auth={map[]} err="requesting tags: errors:\ndenied: requested access to the resource is denied\nunauthorized: authentication required\n"

I'm not sure what the best way to debug this is. I'd like to know if my regcred secrets are even being found. I suspect not based on the auth={map[]}. What would be the next step in debugging this?

[lgebhardt]

Update, I have confirmed my secret's generated auth section is good.

I have confirmed flux is using the secret from the fluxcd namespace since it logs that it's not found when I delete it.

I am still however getting failures:

auth={map[]} err="requesting tags: errors:\ndenied: requested access to the resource is denied\nunauthorized: authentication required\n"

The account at docker hub is linked to an organization. The account used is a member of a team with admin rights in the organization.

Any ideas?

[hiddeco]

@lgebhardt I get the impression that you are attaching the imagePullSecret to Flux. The imagePullSecret should be attached to the workload that makes use of the image, or the service account assigned to the workload, and not Flux itself.

[lgebhardt]

@hiddeco I had set it on the helm-operator like:

helm upgrade -i helm-operator fluxcd/helm-operator \
--namespace fluxcd \
--set image.pullSecret=regcred

When I rebuilt my cluster without that I no longer get the message about the missing regcred, even when they are missing. This is the state I was in when I first made this issue.

How do I attach the imagePullSecret to the workload or service account?

[lgebhardt]

https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#add-imagepullsecrets-to-a-service-account

I got past the auth error by adding the imagePullSecret to the namespace's default serviceaccount.

kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "regcred"}]}' -n stg

The confusing part to me was I thought I needed to do this for the flux serviceaccount, or the flux namespace.

[2opremio]

I suspect this could have been caused by the same problem fixed by #2728 . @lgebhardt could you test whether 2opremio/flux:scope-imagepullsecrets-correctly-a802915a fixes the problem?

[lgebhardt]

@2opremio Sorry for the very late reply. I just rebuilt a cluster with flux 1.19.0 and the issue still seems to be there for me. Without creating the secret in each namespace and patching the service account I can't pull from my private docker repo. I suspect I'm probably missing something obvious.

[Ekion-1]

Having the same issue here. @lgebhardt did you ever solve this?

[SOLDIERz]

This issue still exists. Facing the same Issue with fluxcd in my Kubernetes Cluster right now. Trying to add "imagePullSecrets" in the values section inside the helmrelease, will be completly ignored and "some" default credentials will be used when trying to pull the container.