alerting: provide more examples for alert rules

[kayrus]

I.e. most common (critical and warning severity):

• disk usage
• memory usage
• CPU usage
• kubelet limit for pods per node
• etc.

[beorn7]

There is also http://tips.robustperception.io/ , where concrete examples are supposed to live.

[kayrus]

@beorn7 try to google for "prometheus disk alert example" and you'll find nothing about this page.

Also when I tried to search for "alert" keyword (http://tips.robustperception.io/search:site/q/alert) I got An error occurred when processing your request.

[beorn7]

I'm not saying the solution is already on that site, or that the site is already in perfect operational condition. It's just the thing that @brian-brazil recently set up for Prometheus tips. When addressing the issue you have filed, we should think how many examples we want to add to the core docs (which are supposed to be very concise) and which might be better suited on the tips site.

[brian-brazil]

To be clear I want the reference docs to stay as reference docs, I'm all for different types of docs being on prometheus.io if we can find a sane way to approach it.

[kayrus]

@beorn7 don't think about which ones, just shoot. users need examples. and complete examples, try to avoid taken out of context examples.

[varac]

Coming from nagios, this is what lacks from my perception. I'm totally sold for most parts of prometheus, but I'm missing not only alert examples, I'm missing sane defaults (could be opt-in) for alerts. This would help lowering the bar for ppl who want to migrate from other solutions.

[arj22]

+1

[brancz]

@juliusv has started an initiative to do this and started with metrics from the node_exporter.

[imickeyj]

can anyone give some alert examples or some specific link, I am kind of stuck at setting alerts

[myfreax]

+1

[4220182]

+1

[juliusv]

@imickeyj For now, you can find plenty of alerting rule examples in GitLab's alerting configs: https://gitlab.com/gitlab-com/runbooks/tree/master/alerts

[auhlig]

At SAP we use Prometheus for quite a while now. Some inspiration for alerting rules can be found here.

[tangyong]

@juliusv @auhlig Great sharing!

Still best wish team to add more alerting rules to sources and documents.
Thanks!

[brancz]

We agree with all of this and acknowledge that there is a need for this. The problem today is that there is no and likely will never be a standardized labeling of things, and that's probably a good thing. @tomwilkie and I have been extensively talking about some solutions to this and we hope that soon we will have a proposal out for actually shareable alerting rules and dashboard definitions.

Whether this should be on prometheus.io is in my opinion questionable, my opinion is that the respective "bundle" should be in the repository of the application itself that they describe, similar to how the etcd project has alerting rules and dashboard definitions in the op-guide. As I mentioned the how reusable these actually are today with a lot of assumptions on labeling is questionable, but we are trying to solve that problem.