introducing-prometheus-and-grafana.md

Introducing Prometheus and Grafana

This exercise will demonstrate deploying the Prometheus and Grafana applications using Helm. We will also deploy the sentences application and inspect how metrics are propagated from the application through Prometheus and displayed in Grafana.

Deploying Prometheus and Grafana

First we will install Prometheus and Grafana using Helm. We will install them from their community repositories.

First add repositories to your Helm installation:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add grafana https://grafana.github.io/helm-charts

Next, you can inspect your Helm repositories with helm repo list:

helm repo list
NAME                    URL
prometheus-community    https://prometheus-community.github.io/helm-charts
grafana                 https://grafana.github.io/helm-charts

To look for available Helm chart you can use the helm search feature, e.g.:

helm search repo prometheus
NAME                                                    CHART VERSION   APP VERSION     DESCRIPTION
prometheus-community/kube-prometheus-stack              55.8.1          v0.70.0         kube-prometheus-stack ...
prometheus-community/prometheus                         25.8.2          v2.48.1         Prometheus is a monito...
...

This show that Prometheus chart version 55.8.1 is available and that the version of the Prometheus application in that chart is 2.48.1. You most likely will see newer versions when trying this out...

To install Prometheus and Grafana with settings suitable for the following exercises use the following commands: (note the versions on the commands, this exercise might work with never versions, but for the sake of the exercise, try with these first.)

helm upgrade -i prometheus prometheus-community/prometheus --version 25.8.2 -f resources/values-prometheus.yaml
helm upgrade -i grafana grafana/grafana --version 7.2.1 -f resources/values-grafana.yaml

💡 If you are running on this outside of class:

This exercise assume a Kubernetes cluster with metrics support. Most managed Kubernetes solutions will have support for metrics. On self-deployed clusters one might need to deploy the metrics-server application:

helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server
helm install metrics-server metrics-server/metrics-server --version 3.11.0 --set args[0]="--kubelet-insecure-tls"

After running these command you can inspect the installed Helm-based applications with helm ls:

helm ls
NAME            NAMESPACE        REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
grafana         student-1        1               2024-01-15 15:28:12.704540469 +0100 CET deployed        grafana-7.2.1           10.2.3
prometheus      student-1        1               2024-01-15 15:23:16.442377833 +0100 CET deployed        prometheus-25.8.2       v2.48.1

Also, inspect the PODs that these applications are based upon:

kubectl get pods
grafana-5c7b9b967f-pnkd2             2/2     Running   0          71s
prometheus-server-868b8cdb59-d7gpq   2/2     Running   0          48s

Note that Grafana is deployed without TLS and as such this is not a deployment that is suitable for production use.

Grafana is exposed with a Kubernetes service of type NodePort. Use the following commands to get the external IP address/port and the Grafana admin user password:

kubectl get -o jsonpath="{.spec.ports[0].nodePort}" services grafana
kubectl get nodes -o wide
kubectl get secret grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

Deploying the Application

The sentences application can be deployed with the following command.

kubectl apply -f sentences-app/deploy/kubernetes/

After this, we should have 5 PODs running (Prometheus, Grafana and three PODs for the sentences application). To verify that the application is running, look up the main service and node-port and fetch a result with curl as we did in the hello-sentences-app exercise.

Debugging Metrics

In the following we will look into:

• How to manually query metrics exported from an application prepared for Prometheus
• How Prometheus maintains status for each auto-discovered scrape-targets (like our application PODs)
• How we can query and display metrics with Prometheus, i.e. without Grafana

Going through these steps will not only give good insight into the metrics framework but also show the primary techniques necessary when application metrics fail to be available in Grafana.

Debugging Metrics From PODs

First, lets manually query the metrics from our sentences application. Remember that each microservice exports a metric called sentence_requests_total which is a counter that increments for each request the microservices processes.

The sentences application exports metrics on port 8080 and path '/metrics'. Prometheus can be configured to fetch metrics from any port and path, and we will later see how Prometheus figures out which port to scrape for metrics.

Tasks

• find the nodeport the service is running at

kubectl get svc sentence -o jsonpath="{.spec.ports[0].nodePort}" # Get the node port for the sentence service

• query the sentences service a couple of times to make sure you have some metrics to show

curl -s <NODE-IP>:<PORT> # Get a sentence

• query the metrics endpoint to see the metrics

curl -s <NODE-IP>:<PORT>/metrics | egrep '^sentence'

We use egrep here to filter out the essential information, and we will see the following output:

sentence_requests_total{type="sentence"} 6

💡 I get no results!

If you get no results here, it could be because the sentences application has recently been deployed and metrics with a value of zero are not exported, i.e. until you fetch a sentence for the first time, the curl above will return nothing. Try fetching a sentence as shown in prior exercises and fetch metrics again.

Try fetching more sentences and metrics to see the sentence_requests_total metric increment.

This show that we have a metric with name sentence_requests_total with a label type that have the value sentence. The other microservices age and name use values age and name respectively for the type label. Metrics can have many labels and with these we can separate metrics into different dimensions, e.g. microservice type, versions etc. The metric name sentence_request_total can be considered the what and the labels define the who.

Our curl operation showed that the metrics API on the sentences application works as expected. If this had not worked, we should debug the microservice itself.

Debugging Prometheus Metrics Scraping

Next, we look into Prometheus. The Prometheus GUI is exposed using a NodePort-type service. Look up the node port as we did above with Grafana and open the GUI in a web browser:

kubectl get -o jsonpath="{.spec.ports[0].nodePort}" services prometheus-server

When you have the Prometheus GUI op, select the Status menu item and then the Targets option as shown below. Apply the filter student-X where X is your student number, to show only things from your namespace.

Scroll down to the section with the heading kubernetes-pods (3/3 up)

This shows the list of targets that Prometheus has identified through the Kubernetes API. We see that three POD target show up with the labels matching the three microservices of the sentences application (if you are running the exercises on a shared cluster you may see other users applications as well).

💡 I get no results!

If our list of scrape targets do not match our expectation we should verify that 1) Our Kubernetes resources are correctly annotated to tell Prometheus to scrape then and that the port and path is correct and 2) that Prometheus is deployed with proper Kubernetes authorization to auto-discover scrape targets through the Kubernetes API.

Next, select the Graph menu item and type in sentence_requests_total in the query box as shown below and press return. This will show the number of requests handled by each of the microservices in the sentences application. The fact that we see metrics here verifies that the metrics of the sentences application are available from Prometheus.

Explore Metrics in Grafana

The GUI Prometheus is mostly used for low-level debugging and metrics are typically visualized with Grafana since Grafana provides many more options for building dashboards.

When logged-in to Grafana go to the Explore option in the left-side (see picture below). This feature provide access to a simple metrics query option, which often is used instead of directly accessing Prometheus when we are beyond basic debugging of the metrics feature.

In the query box type the metrics name sentence_requests_total and press the Run Query button in the top-right corner as shown below:

This shows that Grafana can access metrics from Prometheus.

Deploying a Dashboard

Dashboards can be created manually but often an existing dashboard should be used. Grafana supports dynamically loading of dashboards through Kubernetes ConfigMaps. If we have an existing dashboard in JSON format, we can simply create a ConfigMap containing the JSON file and annotate the ConfigMap to let Grafana know it should use the content as a dashboard.

To install a dashboard for the sentences application create and annotate a ConfigMap as follows:

kubectl create configmap dashboard --from-file sentences-app/dashboard-w-ns-selector.json
kubectl label configmap dashboard grafana_dashboard='1'

After this you will be able to use the dashboard in Grafana. Go to the Dashboards option and select Home and/or Manage to see newly loaded dashboards (You might need to reload the Grafana tab first). Note that the dashboard have a namespace selector in the top and you will need to select your own namespace to monitor your sentences application.

To generate some load on the sentences application, use the following command and watch the effect in the dashboard:

kubectl apply -f resources/load-generator.yaml

After a short while, you should see the effect of the generated load in the dashboards with requests/s and also the POD cpu usage.

The time it takes for the data to arrive in Grafana, is dependent on the scrape_interval configured for Prometheus. This can be located in the Prometheus UI under Status -> Configuration.

Autoscaling

if you are doing this exercise after having done the auto-scaling section, you could deploy the HorisontalPODAutoscaler resource to see auto-scaling and the dashboard with POD counts for the individial microservices to change.

kubectl apply -f sentences-app/deploy/hpa.yaml

Cleanup

NB: If you are going to proceed to the next Prometheus/Grafana exercise, you will want to leave everything running for now.

💡 Cleanup for stand-alone use of the exercise

Delete the applications and additional services with the following commands.

kubectl delete -f sentences-app/deploy/kubernetes/
helm delete grafana
helm delete prometheus
kubectl delete configmap dashboard
kubectl delete -f resources/load-generator.yaml
kubectl delete -f sentences-app/deploy/hpa.yaml