SLO and Error Budget monitoring (#1)

Author: danielfm

README.md

@@ -1,15 +1,15 @@
 # Prometheus For Developers
 
-This is an introductory tutorial/workshop I created for telling the software
-developers in my [company](https://descomplica.com.br) the basics about
+This is an introductory tutorial I created for telling the software developers
+in my [company](https://descomplica.com.br) the basics about
 [Prometheus](https://prometheus.io).
 
 If you have any suggestion to improve this content, don't hesitate to contact
 me. Pull Requests are welcome!
 
 ## Table of Contents
 
-- [Workshop Project](#workshop-project)
+- [The Project](#the-project)
   - [Pre-Requisites](#pre-requisites)
   - [Running the Code](#running-the-code)
   - [Cleaning Up](#cleaning-up)
@@ -24,13 +24,14 @@ me. Pull Requests are welcome!
     - [Quantile Estimation Errors](#quantile-estimation-errors)
   - [Measuring Throughput](#measuring-throughput)
   - [Measuring Memory/CPU Usage](#measuring-memorycpu-usage)
+  - [Measuring SLOs and Error Budgets](#measuring-slos-and-error-budgets)
   - [Monitoring Applications Without a Metrics Endpoint](#monitoring-applications-without-a-metrics-endpoint)
   - [Final Gotchas](#final-gotchas)
 - [References](#references)
 
-## Workshop Project
+## The Project
 
-This workshop follows a more practical approach (with hopefully just the
+This tutorial follows a more practical approach (with hopefully just the
 right amount of theory!), so we provide a simple Docker Compose configuration
 for simplifying the project bootstrap.
 
@@ -311,29 +312,27 @@ to the correct receiver integration (i.e. email, Slack, PagerDuty,
 OpsGenie). It also takes care of silencing and inhibition of alerts.
 
 Configuring Alertmanager to send metrics to PagerDuty, or Slack, or whatever,
-is out of the scope of this workshop, but we can still play around with alerts.
+is out of the scope of this tutorial, but we can still play around with alerts.
 
-Let's define our first alerting rule in
+We already have the following alerting rule defined in
 `config/prometheus/prometheus.rules.yml`:
 
 ```yaml
-# Uptime alerting rule
 groups:
-- name: uptime
-  rules:
-  - alert: ServerDown
-    expr: up == 0
-    for: 1m
-    labels:
-      severity: page
-    annotations:
-      summary: One or more targets are down
-      description: Instance {{ $labels.instance }} of {{ $labels.job }} is down
+  - name: uptime
+    rules:
+      # Uptime alerting rule
+      # Ref: https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/
+      - alert: ServerDown
+        expr: up == 0
+        for: 1m
+        labels:
+          severity: page
+        annotations:
+          summary: One or more targets are down
+          description: Instance {{ $labels.instance }} of {{ $labels.job }} is down
 ```
 
-Restart Prometheus with `docker-compose restart prometheus` and open the Alerts
-page at <http://localhost:9090/alerts>.
-
 ![Prometheus alerts](./img/prometheus-alerts-1.png)
 
 Each alerting rule in Prometheus is also a time series, so in this case you can
@@ -363,7 +362,7 @@ should go back to a green state after a few seconds.
 
 ## Instrumenting Your Applications
 
-Let's examine a sample Node.js application we created for this workshop.
+Let's examine a sample Node.js application we created for this tutorial.
 
 Open the `./sample-app/index.js` file in your favorite text editor. The
 code is fully commented, so you should not have a hard time understanding
@@ -373,15 +372,21 @@ it.
 
 We can measure request durations with
 [percentiles](https://en.wikipedia.org/wiki/Quantile) or
-[averages](https://en.wikipedia.org/wiki/Arithmetic_mean). However,
-it's not recommended relying on averages to track request durations because
-averages can be very misleading (see the [References](#references) for a few
-posts on the pitfalls of averages and how percentiles can help).
+[averages](https://en.wikipedia.org/wiki/Arithmetic_mean). It's not
+recommended relying on averages to track request durations because averages
+can be very misleading (see the [References](#references) for a few posts on
+the pitfalls of averages and how percentiles can help). A better way for
+measuring durations is via percentiles as it tracks the user experience
+more closely:
+
+![Percentiles as a way to measure user satisfaction](./img/percentiles.jpg)
+Source: [Twitter](https://twitter.com/rakyll/status/1045075510538035200)
 
 In Prometheus, we can generate percentiles with summaries or histograms.
 
 To show the differences between these two, our sample application exposes
-two custom metrics for measuring request durations with:
+two custom metrics for measuring request durations with, one via a summary
+and the other via a histogram:
 
 ```js
 // Summary metric for measuring request durations
@@ -394,7 +399,7 @@ const requestDurationSummary = new prometheusClient.Summary({
 
   // Extra dimensions, or labels
   // HTTP method (GET, POST, etc), and status code (200, 500, etc)
-  labelNames: ['method', 'statuscode'],
+  labelNames: ['method', 'status'],
 
   // 50th (median), 75th, 90th, 95th, and 99th percentiles
   percentiles: [0.5, 0.75, 0.9, 0,95, 0.99]
@@ -410,19 +415,19 @@ const requestDurationHistogram = new prometheusClient.Histogram({
 
   // Extra dimensions, or labels
   // HTTP method (GET, POST, etc), and status code (200, 500, etc)
-  labelNames: ['method', 'statuscode'],
+  labelNames: ['method', 'status'],
 
   // Duration buckets, in seconds
   // 5ms, 10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 2.5s, 5s, 10s
   buckets: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]
 });
 ```
 
-As you can see, in a summary we specify the percentiles in which we
-want the Prometheus client to calculate and report latencies, while in a
-histogram we specify the buckets where the observed durations will be
-classified (i.e. a 300ms duration will be stored in the 250ms-500ms
-bucket).
+As you can see, in a summary we specify the percentiles in which we want the
+Prometheus client to calculate and report latencies for, while in a histogram
+we specify the duration buckets in which the observed durations will be stored
+as a counter (i.e. a 300ms observation will be stored by incrementing the
+counter corresponding to the 250ms-500ms bucket).
 
 Our sample application introduces a one-second delay in approximately 5%
 of requests, just so we can compare the average response time with
@@ -470,7 +475,7 @@ rate(sample_app_summary_request_duration_seconds_sum[15s]) / rate(sample_app_sum
 sample_app_summary_request_duration_seconds{quantile="0.99"}
 
 # 99th percentile (via histogram)
-histogram_quantile(0.99, sum(rate(sample_app_histogram_request_duration_seconds_bucket[15s])) by (le, method, statuscode))
+histogram_quantile(0.99, sum(rate(sample_app_histogram_request_duration_seconds_bucket[15s])) by (le, method, status))
 ```
 
 The result of these queries may seem surprising.
@@ -501,7 +506,7 @@ Quoting the [documentation]():
 > `histogram_quantile()` function.
 
 In other words, for the quantile estimation from the buckets of a
-histogram to be accurate, we need to be careful when picking the bucket
+histogram to be accurate, we need to be careful when choosing the bucket
 layout; if it doesn't match the range and distribution of the actual
 observed durations, you will get inaccurate quantiles as a result.
 
@@ -518,7 +523,7 @@ const requestDurationHistogram = new prometheusClient.Histogram({
 
   // Extra dimensions, or labels
   // HTTP method (GET, POST, etc), and status code (200, 500, etc)
-  labelNames: ['method', 'statuscode'],
+  labelNames: ['method', 'status'],
 
   // Duration buckets, in seconds
   // 5ms, 10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 2.5s, 5s, 10s
@@ -527,14 +532,14 @@ const requestDurationHistogram = new prometheusClient.Histogram({
 ```
 
 Here we are using a _exponential_ bucket configuration in which the buckets
-get larger as latency gets bigger. This is a widely used pattern; since we
+double in size at every step. This is a widely used pattern; since we
 always expect our services to respond quickly (i.e. with response time
 between 0 and 300ms), we specify more buckets for that range, and fewer
 buckets for request durations we think are less likely to occur.
 
 According to the previous plot, all slow requests from our application
-is falling into the 1s-2.5s bucket, causing us to lose precision when
-calculating the 99th percentile.
+are falling into the 1s-2.5s bucket, resulting in this loss of precision
+when calculating the 99th percentile.
 
 Since we know our application will take at most ~1s to respond, we can
 choose a more appropriate bucket layout:
@@ -573,10 +578,10 @@ The reason is efficiency. Remember:
 
 **more buckets == more time series == more space == slower queries**
 
-Let's say you have an SLA to serve 99% of requests within 300ms. If all
-you want to know is whether you are honoring your SLA or not, it doesn't
-really matter if the quantile estimation is not accurate for requests
-slower than 300ms.
+Let's say you have an SLO (more details on SLOs later) to serve 99% of
+requests within 300ms. If all you want to know is whether you are
+honoring your SLO or not, it doesn't really matter if the quantile
+estimation is not accurate for requests slower than 300ms.
 
 You might also be wondering: if summaries are more precise, why not use
 summaries instead of histograms?
@@ -593,7 +598,8 @@ with multiple replicas, you can safely use the `histogram_quantile()`
 function to calculate the 99th percentile across all requests to all
 replicas. You cannot do this with summaries. I mean, you can `avg()` the
 99th percentiles of all replicas, or take the `max()`, but the value you
-get will be statistically incorrect.
+get will be statistically incorrect and could not be used as a proxy to the
+99th percentile.
 
 ---
 
@@ -652,6 +658,108 @@ Grafana server at <http://localhost:3000>.
 
 ---
 
+### Measuring SLOs and Error Budgets
+
+> Managing service reliability is largely about managing risk, and managing risk
+> can be costly.
+>
+> 100% is probably never the right reliability target: not only is it impossible
+> to achieve, it's typically more reliability than a service's users want or
+> notice.
+
+SLOs, or _Service Level Objectives_, is one of the main tools employed by
+[Site Reliability Engineers (SREs)](https://landing.google.com/sre/books/) for
+making data-driven decisions about reliability.
+
+SLOs are based on SLIs, or _Service Level Indicators_, which are the key metrics
+that define how well (or how poorly) a given service is operating. Common SLIs
+would be the number of failed requests, the number of requests slower than some
+threshold, etc. Although different types of SLOs can be useful for different
+types of systems, most HTTP-based services will have SLOs that can be
+classified into two categories: **availability** and **latency**.
+
+For instance, let's say these are the SLOs for our sample application:
+
+| Category | SLI | SLO |
+|-|-|-|
+| Availability | The proportion of successful requests; any HTTP status other than 500-599 is considered successful | 95% successful requests |
+| Latency      | The proportion of requests with duration less than or equal to 100ms | 95% requests under 100ms |
+
+The difference between 100% and the SLO is what we call the _Error Budget_.
+In this example, the error budget for both SLOs is 5%; if the application
+receives 1,000 requests during the SLO window (let's say one minute for the
+purposes of this tutorial), it means that 50 requests can fail and we'll
+still meet our SLO.
+
+But do we need additional metrics for keeping track of these SLOs? Probably
+not. If you are tracking request durations with a histogram (as we are here),
+chances are you don't need to do anything else. You already got all the
+metrics you need!
+
+Let's send a few requests to the server so we can play around with the metrics:
+
+```sh
+$ while true; do curl -s http://localhost:4000 > /dev/null ; done
+```
+
+```sh
+# Number of requests served in the SLO window
+sum(increase(sample_app_histogram_request_duration_seconds_count[1m])) by (job)
+
+# Number of requests that violated the latency SLO (all requests that took more than 100ms to be served)
+sum(increase(sample_app_histogram_request_duration_seconds_count[1m])) by (job) - sum(increase(sample_app_histogram_request_duration_seconds_bucket{le="0.1"}[1m])) by (job)
+
+# Number of requests in the error budget: (100% - [slo threshold]) * [number of requests served]
+(1 - 0.95) * sum(increase(sample_app_histogram_request_duration_seconds_count[1m])) by (job)
+
+# Remaining requests in the error budget: [number of requests in the error budget] - [number of requests that violated the latency SLO]
+(1 - 0.95) * sum(increase(sample_app_histogram_request_duration_seconds_count[1m])) by (job) - (sum(increase(sample_app_histogram_request_duration_seconds_count[1m])) by (job) - sum(increase(sample_app_histogram_request_duration_seconds_bucket{le="0.1"}[1m])) by (job))
+
+# Remaining requests in the error budget as a ratio: ([number of requests in the error budget] - [number of requests that violated the SLO]) / [number of requests in the error budget]
+((1 - 0.95) * sum(increase(sample_app_histogram_request_duration_seconds_count[1m])) by (job) - (sum(increase(sample_app_histogram_request_duration_seconds_count[1m])) by (job) - sum(increase(sample_app_histogram_request_duration_seconds_bucket{le="0.1"}[1m])) by (job))) / ((1 - 0.95) * sum(increase(sample_app_histogram_request_duration_seconds_count[1m])) by (job))
+```
+
+Due to the simulated scenario in which ~5% of requests takes 1s to complete,
+if you try the last query you should see that the average budget available
+is around 0%, that is, we have no more budget to spend and will inevitably
+break the latency SLO if more requests start to take more time to be served.
+This is not a good place to be.
+
+![Error Budget Burn Rate of 1x](./img/slo-1.png)
+
+But what if we had a more strict SLO, say, 99% instead of 95%? What would be
+the impact of these slow requests on the error budget?
+
+Just replace all `0.95` by `0.99` in that query to see what would happen:
+
+![Error Budget Burn Rate of 3x](./img/slo-2.png)
+
+In the previous scenario with the 95% SLO, the SLO _burn rate_ was ~1x, which
+means the whole error budget was being consumed during the SLO window, that is,
+in 60 seconds. Now, with the 99% SLO, the burn rate was ~3x, which means that
+instead of taking one minute for the error budget to exhaust, it now takes
+only ~20 seconds!
+
+Now change the `curl` to point to the `/metrics` endpoint, which do not have
+the simulated long latency for 5% of the requests, and you should see the error
+budget go back to 100% again:
+
+```bash
+$ while true; do curl -s http://localhost:4000/metrics > /dev/null ; done
+```
+
+![Error Budget Replenished](./img/slo-3.png)
+
+---
+
+**Want to know more?** The
+[Site Reliability Workbook](https://landing.google.com/sre/books/) is a great
+resource on this topic and includes more advanced concepts such as how to alert
+based on SLO burn rate as a way to improve alert precision/recall and
+detection/reset times.
+
+---
+
 ### Monitoring Applications Without a Metrics Endpoint
 
 We learned that Prometheus needs all applications to expose a `/metrics`
@@ -688,3 +796,4 @@ hard time when creating queries later.
 - [Blog Post: Understanding Machine CPU usage](https://www.robustperception.io/understanding-machine-cpu-usage/)
 - [Blog Post: #LatencyTipOfTheDay: You can't average percentiles. Period.](http://latencytipoftheday.blogspot.com/2014/06/latencytipoftheday-you-cant-average.html)
 - [Blog Post: Why Averages Suck and Percentiles are Great](https://www.dynatrace.com/news/blog/why-averages-suck-and-percentiles-are-great/)
+- [Site Reliability Engineering books](https://landing.google.com/sre/books/)