The Zen of Prometheus #1783

kakkoyun · 2020-11-04T10:24:30Z

Signed-off-by: Kemal Akkoyun kakkoyun@gmail.com

As suggested and discussed on #1692, I would like to make The Zen of Prometheus an official part of documentation.

It would be great if you could help me to refine it by your reviews
@brian-brazil @beorn7 @SuperQ @juliusv @brancz @bwplotka @RichiH

Fix #1692

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

brian-brazil · 2020-11-04T10:35:41Z

content/docs/practices/the_zen.md

+
+## Instrument first, ask questions later
+
+During development you will never know what questions you need to ask later. Software needs good instrumentation, it's not optional. Metrics are cheap. Use them generously.


A metric is cheap. Metrics are not cheap.

Metrics without labels are cheap, metrics with labels tend to blow up in cardinality. Making it seem that metrics are cheap would likely lead to more users falling into this trap.

Yeah, would be good to be more nuanced here and perhaps link to the cardinality section.

brian-brazil · 2020-11-04T10:36:53Z

content/docs/practices/the_zen.md

+
+## Measure what users care
+
+Do your users care if your database servers are down? Do they care about your CPU saturation? No, they care about what they experience. They care about whether they can access the page that they have requested and their results are fresh. Think in terms of latencies and availability. Let your [SLO](https://landing.google.com/sre/sre-book/chapters/service-level-objectives/)s guide your instrumentation.


Uhm, you should be measuring all of those things.

You're conflating metrics with alerting here.

I think this should be phrased in a way that measuring all of those things is still important, but that instrumentation should additionally be suitable for SLO-based alerting.

brian-brazil · 2020-11-04T10:37:25Z

content/docs/practices/the_zen.md

+
+## Avoid missing metrics
+
+Time series that are not present until something happens are difficult to deal with. To avoid this, export 0 (or NaN if 0 would be misleading) for any time series you know may exist in advance. You have to initialize your metrics with the zero values to prevent broken dashboards and misfiring alerts. For more detailed explanation check out [Existential issues with metrics](https://www.robustperception.io/existential-issues-with-metrics).


NaN shouldn't be used like this.

You don't have to expose 0, but it's much tricker to deal with correctly if you don't.

brian-brazil · 2020-11-04T10:38:19Z

content/docs/practices/the_zen.md

+
+## Cardinality Matters
+
+Every unique set of labels create a new timeseries. Use labels with care watch out what you put into your labels. Avoid cardinality explosion, unbounded labels will blow up Prometheus. And keep in mind that labels are multiplicative. You will have multiple labels, multiple target labels and targets.


The number of target labels doesn't matter for cardinality.

brian-brazil · 2020-11-04T10:39:18Z

content/docs/practices/the_zen.md

+
+## Naming is hard
+
+One of the hardest problem in computer science. Metric name should only have a single meaning within a single job. And you should always guard against metric name collisions among the jobs on your system.


A metric name

What do you mean by "guard against"? It's perfectly normal for different jobs to share metric names, such as process_cpu_seconds_total. As worded this seems to be arguing for prefixing metric names with job names.

brian-brazil · 2020-11-04T10:41:35Z

content/docs/practices/the_zen.md

+
+And as always, let your [SLO](https://www.youtube.com/watch?v=X99X-VDzxnw)s guide your bucket layout, create boundaries to match your SLO.
+
+The histograms underneath are just counter with labels; where bucket boundaries used as labels. Be precautious while adding additional labels to your histograms. Remember *Labels are multiplicative* and [Cardinality Matters](#cardinality-matters).


brian-brazil · 2020-11-04T10:42:00Z

content/docs/practices/the_zen.md

+
+## If you can graph it, you can alert on it
+
+You can't look at dashboards 24/7. Prometheus unified metrics, dashboarding, and alerting. PromQL is the core of every Prometheus alert, and a PromQL query is the source of any graph on a dashboard. That is very powerful. Use it.


unifies

I doubt we were first.

brian-brazil · 2020-11-04T10:42:18Z

content/docs/practices/the_zen.md

+
+## If you run it, then you should put an alert on it
+
+Always have an alert -*at least*- on presence and healthiness of your targets. You can't that rely on the things and take action that you can't observe.


brian-brazil · 2020-11-04T10:42:32Z

content/docs/practices/the_zen.md

+
+Always have an alert -*at least*- on presence and healthiness of your targets. You can't that rely on the things and take action that you can't observe.
+
+Avoid missing targets and unhealthy targets. All of the client libraries provides `up` metric by default. Use it.


Client libraries do not produce up, that comes from Prometheus.

brian-brazil · 2020-11-04T10:43:55Z

content/docs/practices/the_zen.md

+
+## Please five more minutes
+
+Prometheus alerting rules let you to specify a time duration that determines after how long an alert should start firing according to given query, which is called `FOR`. If you don't specify it a single failed scrape could cause an alert to fire. You need more tolerance. Don't make it short and always specify it. And also don't make it too long. For more information check out [Setting Thresholds on Alerts](https://www.robustperception.io/setting-thresholds-on-alerts).


It's just for these days.

I'd recommend simplifying, and saying always at least 5 minutes

I think "always" is too strong. I would word it more along the lines that you should only do something else if you know exactly what you are doing.

beorn7

I like the idea of adding this here a lot.

Of course, the original idea had a humorous touch and was meant to be taken with a grain of salt. As an "official" statement here on the project's website it will receive quite a lot of scrutiny…

beorn7 · 2020-11-04T14:29:03Z

content/docs/practices/the_zen.md

+
+## Instrument first, ask questions later
+
+During development you will never know what questions you need to ask later. Software needs good instrumentation, it's not optional. Metrics are cheap. Use them generously.


Yeah, would be good to be more nuanced here and perhaps link to the cardinality section.

beorn7 · 2020-11-04T14:29:55Z

content/docs/practices/the_zen.md

+
+The First and the most important rule, if you have to remember only one thing remember this one. Instrument all the things!
+
+## Measure what users care


… care about?

beorn7 · 2020-11-04T14:30:41Z

content/docs/practices/the_zen.md

+
+## Measure what users care
+
+Do your users care if your database servers are down? Do they care about your CPU saturation? No, they care about what they experience. They care about whether they can access the page that they have requested and their results are fresh. Think in terms of latencies and availability. Let your [SLO](https://landing.google.com/sre/sre-book/chapters/service-level-objectives/)s guide your instrumentation.


I think this should be phrased in a way that measuring all of those things is still important, but that instrumentation should additionally be suitable for SLO-based alerting.

beorn7 · 2020-11-04T14:31:40Z

content/docs/practices/the_zen.md

+
+---
+
+## Counters rule and gauges suck


It's not a nuanced statement, it's a proverb!

It refers to not using gauges for something that can be represented as a counter. That's explained in the text below.

beorn7 · 2020-11-04T14:33:26Z

content/docs/practices/the_zen.md

+
+## Please five more minutes
+
+Prometheus alerting rules let you to specify a time duration that determines after how long an alert should start firing according to given query, which is called `FOR`. If you don't specify it a single failed scrape could cause an alert to fire. You need more tolerance. Don't make it short and always specify it. And also don't make it too long. For more information check out [Setting Thresholds on Alerts](https://www.robustperception.io/setting-thresholds-on-alerts).


I think "always" is too strong. I would word it more along the lines that you should only do something else if you know exactly what you are doing.

Initialize The Zen of Prometheus

0c486a9

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

brian-brazil reviewed Nov 4, 2020

View reviewed changes

beorn7 requested changes Nov 4, 2020

View reviewed changes


		## Instrument first, ask questions later

		During development you will never know what questions you need to ask later. Software needs good instrumentation, it's not optional. Metrics are cheap. Use them generously.


		## Measure what users care

		Do your users care if your database servers are down? Do they care about your CPU saturation? No, they care about what they experience. They care about whether they can access the page that they have requested and their results are fresh. Think in terms of latencies and availability. Let your [SLO](https://landing.google.com/sre/sre-book/chapters/service-level-objectives/)s guide your instrumentation.


		## Avoid missing metrics

		Time series that are not present until something happens are difficult to deal with. To avoid this, export 0 (or NaN if 0 would be misleading) for any time series you know may exist in advance. You have to initialize your metrics with the zero values to prevent broken dashboards and misfiring alerts. For more detailed explanation check out [Existential issues with metrics](https://www.robustperception.io/existential-issues-with-metrics).


		## Cardinality Matters

		Every unique set of labels create a new timeseries. Use labels with care watch out what you put into your labels. Avoid cardinality explosion, unbounded labels will blow up Prometheus. And keep in mind that labels are multiplicative. You will have multiple labels, multiple target labels and targets.


		## Naming is hard

		One of the hardest problem in computer science. Metric name should only have a single meaning within a single job. And you should always guard against metric name collisions among the jobs on your system.


		And as always, let your [SLO](https://www.youtube.com/watch?v=X99X-VDzxnw)s guide your bucket layout, create boundaries to match your SLO.

		The histograms underneath are just counter with labels; where bucket boundaries used as labels. Be precautious while adding additional labels to your histograms. Remember Labels are multiplicative and [Cardinality Matters](#cardinality-matters).


		## If you can graph it, you can alert on it

		You can't look at dashboards 24/7. Prometheus unified metrics, dashboarding, and alerting. PromQL is the core of every Prometheus alert, and a PromQL query is the source of any graph on a dashboard. That is very powerful. Use it.


		## If you run it, then you should put an alert on it

		Always have an alert -at least- on presence and healthiness of your targets. You can't that rely on the things and take action that you can't observe.


		Always have an alert -at least- on presence and healthiness of your targets. You can't that rely on the things and take action that you can't observe.

		Avoid missing targets and unhealthy targets. All of the client libraries provides `up` metric by default. Use it.


		## Please five more minutes

		Prometheus alerting rules let you to specify a time duration that determines after how long an alert should start firing according to given query, which is called `FOR`. If you don't specify it a single failed scrape could cause an alert to fire. You need more tolerance. Don't make it short and always specify it. And also don't make it too long. For more information check out [Setting Thresholds on Alerts](https://www.robustperception.io/setting-thresholds-on-alerts).


		The First and the most important rule, if you have to remember only one thing remember this one. Instrument all the things!

		## Measure what users care

The Zen of Prometheus #1783

Are you sure you want to change the base?

The Zen of Prometheus #1783

Uh oh!

Conversation

kakkoyun commented Nov 4, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

beorn7 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kakkoyun commented Nov 4, 2020 •

edited

Loading