Grafana-operator creates many versions of dashboards

[asahnovskiy-deloitte]

Describe the bug
The grafana-operator create new version of dashbord every 5 - 10 muntes after creating dashboards from crd

Version
v5.0.0-rc3

To Reproduce
Steps to reproduce the behavior:

apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
  name: grafana
  namespace: dev
  labels:
    dashboards: "grafana-dev"
spec:
  config:
    log:
      mode: "console"
  deployment:
    spec:
      template:
        metadata:
          annotations:
            sidecar.istio.io/inject: "false"
          labels:
            test: "test"
        spec:
          securityContext:
            fsGroup: 472
            runAsGroup: 472
            runAsNonRoot: true
            runAsUser: 472
          containers:
            - name: grafana
              image: grafana/grafana:9.4.3
              securityContext:
                allowPrivilegeEscalation: true
                readOnlyRootFilesystem: false
              env:
                - name: GF_ALERTING_ENABLED
                  value: "false"
                - name: GF_AUTH_ANONYMOUS_ENABLED
                  value: "false"
                - name: GF_AUTH_DISABLE_LOGIN_FORM
                  value: "false"
                - name: GF_SECURITY_ADMIN_USER
                  valueFrom:
                    secretKeyRef:
                      key: admin-user
                      name: grafana-admin-creds
                - name: GF_SECURITY_ADMIN_PASSWORD
                  valueFrom:
                    secretKeyRef:
                      key: admin-password
                      name: grafana-admin-creds
              volumeMounts:
              - mountPath: /var/lib/grafana
                name: storage
          volumes:
            - name: storage
              persistentVolumeClaim:
                claimName: grafana
---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDatasource
metadata:
  name: test-2
  namespace: dev
spec:
  instanceSelector:
    matchLabels:
      dashboards: "grafana-dev"
  datasource:
    name: prometheus
    type: prometheus
    access: proxy
    orgId: 1
    isDefault: true
    url: http://prometheus-kube.monitoring.svc.cluster.local:9090
    editable: true
---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: nginx-ingress
  namespace: dev
spec:
  datasources:
    - inputName: "DS_PROMETHEUS"
      datasourceName: "Prometheus"
  instanceSelector:
    matchLabels:
      dashboards: "grafana-dev"
  grafanaCom:
    id: 9614

[nissessenap]

This was a bug in v5.0.0-rc3, please bump to v5.0.0 and this should be fixed.

[asahnovskiy-deloitte]

@nissessenap I redeployed from scratch with version v5.0.0
The issue still present.

spec:
      containers:
      - args:
        - --health-probe-bind-address=:8081
        - --metrics-bind-address=0.0.0.0:9090
        env:
        - name: WATCH_NAMESPACE
        image: ghcr.io/grafana-operator/grafana-operator:v5.0.0
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 8081
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1

[asahnovskiy-deloitte]

I see every 5-10 minute the new version of dashboards.
I tested a couple of dashboards

[nissessenap]

Okay, can you please share what the logs say?
Is this something that happens for all your dashboards?
Also, what happens if you instead of using grafana com ID download the json manually and upload it to your CRD, do you get the same issue?
If that is the case, what happens if you first import the grafana json in to grafana and then export it again, then take that json and put it in to a CRD?

The reason why I'm asking is that grafana have a bunch of magic to solve breaking changes that remakes the json to new versions. But if they json generated by grafana diffs from the dashboard that you send in, the operator will complain about and try to resync.

[asahnovskiy-deloitte]

Example of logs

1.6868149771189666e+09	INFO	GrafanaDashboardReconciler	found matching Grafana instances for dashboard	{"controller": "grafanadashboard", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDashboard", "GrafanaDashboard": {"name":"nginx-ingress","namespace":"ckarwicki"}, "namespace": "ckarwicki", "name": "nginx-ingress", "reconcileID": "350f4d81-8feb-4805-8618-63ed98a28492", "count": 1}
1.6868149775721378e+09	INFO	GrafanaDashboardReconciler	found matching Grafana instances for dashboard	{"controller": "grafanadashboard", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDashboard", "GrafanaDashboard": {"name":"cert-manager","namespace":"ckarwicki"}, "namespace": "ckarwicki", "name": "cert-manager", "reconcileID": "a703d7d3-d9cb-4d2d-a848-7bb858e521b0", "count": 1}
1.686814977771143e+09	INFO	GrafanaDashboardReconciler	found matching Grafana instances for dashboard	{"controller": "grafanadashboard", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDashboard", "GrafanaDashboard": {"name":"cert-manager","namespace":"ckarwicki"}, "namespace": "ckarwicki", "name": "cert-manager", "reconcileID": "adfe9a11-3b90-499a-9d35-41db0192ebc6", "count": 1}
1.686814979060177e+09	INFO	GrafanaDashboardReconciler	found matching Grafana instances for dashboard	{"controller": "grafanadashboard", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDashboard", "GrafanaDashboard": {"name":"external-dns","namespace":"ckarwicki"}, "namespace": "ckarwicki", "name": "external-dns", "reconcileID": "410ebe3b-84a5-4fa5-bfdd-dbf12a8d6ba6", "count": 1}
1.6868149792541118e+09	INFO	GrafanaDashboardReconciler	found matching Grafana instances for dashboard	{"controller": "grafanadashboard", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDashboard", "GrafanaDashboard": {"name":"external-dns","namespace":"ckarwicki"}, "namespace": "ckarwicki", "name": "external-dns", "reconcileID": "ae6f3c13-30c5-412a-bea6-3d64ddc13db4", "count": 1}

[weisdd]

@asahnovskiy-deloitte This behaviour is expected. There’s periodic resync process in place that is needed to cope with dashboard spec drift (e.g. due to users changing panels through UI), and calculating diffs in this case would have been over-engineering (keys can come in different order with different indentation => everything needs to be represented in the same way before “original” dashboard and the one that exists in Grafana can be compared), so the operator simply re-uploads the dashboard. The default interval is 5 minutes, though you’re free to choose any other number or even disable resync (docs).