[prometheus-kube-stack] Prometheus does not monitor Grafana service

[philippemerle]

Describe the bug a clear and concise description of what the bug is.

Prometheus does not monitor Grafana service

What's your helm version?

ersion.BuildInfo{Version:"v3.16.4", GitCommit:"7877b45b63f95635153b29a42c0c2f4273ec45ca", GitTreeState:"clean", GoVersion:"go1.22.7"}

What's your kubectl version?

Client Version: v1.32.2 Kustomize Version: v5.5.0

Which chart?

kube-prometheus-stack

What's the chart version?

76.2.0

What happened?

It seems that Prometheus does not monitor Grafana service as illustrated in both following diagrams.

Following diagram shows the architecture of Kube Prometheus Stack Helm Chart:

Following diagram shows the architecture of a Kube Prometheus Stack deployed on minikube:

On both diagrams, we could see that the Prometheus resource is connected to all ServiceMonitor resources except for the kube-prometheus-stack-grafana service monitor.

What you expected to happen?

The Prometheus resource should be connected to the kube-prometheus-stack-grafana service monitor. More precisely the kube-prometheus-stack-grafana service monitor should match the labels defined in the spec.serviceMonitorSelector field of the Prometheus resource.

How to reproduce it?

Both diagrams were generated with KubeDiagrams and are available here.

Enter the changed values of values.yaml?

NONE

Enter the command that you execute and failing/misfunctioning.

helm-diagrams -c monitoring.coreos.com.kdc https://prometheus-community.github.io/helm-charts/kube-prometheus-stack

Anything else we need to know?

No response

[philippemerle]

At https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/templates/prometheus/prometheus.yaml#L190, the default value of the serviceMonitorSelector field of the Prometheus resource is set to

  serviceMonitorSelector:
    matchLabels:
      release: {{ $.Release.Name | quote }}

But all Grafana resources are labelled with app.kubernetes.io/instance instead of release. So, the kube-prometheus-stack-grafana service monitor could never match the labels defined in the spec.serviceMonitorSelector field of the Prometheus resource.

To resolve this issue, replace https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/templates/prometheus/prometheus.yaml#L192:

      release: {{ $.Release.Name | quote }}

by

      app.kubernetes.io/instance: {{ $.Release.Name | quote }}

[philippemerle]

Following diagram shows the architecture of the Kube Prometheus Stack Helm Chart after modifying the spec.serviceMonitorSelector field of the Prometheus resource: