Argo .metadata.annotations accessor error in 2.8

[agaudreault]

Describe the bug

After upgrading Argo from 2.5.x -> 2.8.0, I get the following error in my self-managed ArgoCD app. I am using the Argo helm chart 5.34.6, but it does not seem to be the issue. I tried to resync/refresh/hard refresh/invalidate cluster cache/restart redis.

ComparisonError: rpc error: code = Unknown desc = Manifest generation error (cached): .metadata.annotations accessor error: <nil> is of the type <nil>, expected map[string]interface{}

Expected behavior

Successful sync

Screenshots

Version

2.8.0

[agaudreault]

Comes from annotations: being in the Helm template without any value. This is the case for a few resources.

# Source: argocd/charts/argo-cd/templates/argocd-server/servicemonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: argocd-server
  namespace: "argocd"
  labels:
    helm.sh/chart: argo-cd-5.34.6
    app.kubernetes.io/name: argocd-server
    app.kubernetes.io/instance: argocd
    app.kubernetes.io/component: server
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/part-of: argocd
  annotations:     # here, ArgoCD only supports `{}`
spec:
...

Can be mitigated by setting values in the chart such as

  server:
    metrics:
      serviceMonitor:
        annotations:
          fix: 'https://github.com/argoproj/argo-cd/issues/14336'

[crenshaw-dev]

Yep, that matches what I've seen in another case. Not sure what changed causing Argo to no longer handle this well.

[Raziel9572]

I'm trying argocd 2.8-rc2 on sandbox environment. I have multiple applications that have this error message. Do I need to set the annotation on every application or is there something I can do on argocd side to fix this for every application ?

[crenshaw-dev]

@Raziel9572 for now you'll have to change the manifests. But it is a regression, we need to handle this case in Argo CD.

[ohauer]

I think the issue will hit many charts with no values assigned to labels or annotations

• argocd: v2.8.0-rc3
• chart: bitnami/external-dns 6.20.4

will result in yaml with empty annotation for the ServiceAccount and Deployment unless annotation is set explicit in values.yaml

unluckily I found many many more such charts, so I guess reverting back to argocd v2.7 will be the quickest option for me at the moment + test all charts before on a test cluster with argocd-v2.8.x

apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-dns
  labels:
  . app.kubernetes.io/name: external-dns
...
  annotations:  <-- per default no values
automountServiceAccountToken: true

[agaudreault]

@crenshaw-dev Is there a label to identify this bug as a blocker for 2.8?

[crenshaw-dev]

@agaudreault-jive I've added it to the roadmap, such is what I've been using to track blockers.

[agaudreault]

Seems to be caused by #13743.
Error is thrown at line

argo-cd/util/kube/kube.go

Line 25 in 4ed611c

labels, _, err := unstructured.NestedStringMap(target.Object, "metadata", "labels")

depending on the ArgoCD tracking method.