Skip to content

Commit

Permalink
Docs: document creating alert rules (grafana#34839)
Browse files Browse the repository at this point in the history
  • Loading branch information
domasx2 authored May 28, 2021
1 parent 38147cc commit d6037a4
Show file tree
Hide file tree
Showing 5 changed files with 173 additions and 2 deletions.
3 changes: 2 additions & 1 deletion docs/sources/alerting/unified-alerting/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,8 @@ Alerts have four main components:

You can perform the following tasks for alerts:

- [Create an alert rule]({{< relref "./create-alert-rule.md" >}})
- [Create a Grafana managed alert rule]({{< relref "./create-grafana-managed-rule.md" >}})
- [Create a Cortex or Loki managed alert rule]({{< relref "./create-cortex-loki-managed-rule.md" >}})
- [View existing alert rules and their current state]({{< relref "./view-alerts.md" >}})
- [Test alert rules and troubleshoot]({{< relref "./troubleshoot-alerts.md" >}})
- [Add or edit an alert contact point]({{< relref "./contact-points.md" >}})
Expand Down
Empty file.
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
+++
title = "Create Cortex or Loki managed alert rule"
description = "Create Cortex or Loki managed alerting rule"
keywords = ["grafana", "alerting", "guide", "rules", "create"]
weight = 400
+++

# Create a Cortex or Loki managed alerting rule

Grafana allows you manage alerting rules for an external Cortex or Loki instance.

## Add or edit a Cortex or Loki managed alerting rule

1. In the Grafana menu hover your cursor over the Alerting (bell) icon.
1. To create a new alert rule, click **New alert rule**. To edit an existing rule, expand one of the rules in the **Cortex / Loki** section and click **Edit**.
1. Click on the **Alert type** drop down and select **Cortex / Loki managed alert**.
1. Fill out the rest of the fields. Descriptions are listed below in [Alert rule fields](#alert-rule-fields).
1. When you have finished writing your rule, click **Save** in the upper right corner to save the rule,, or **Save and exit** to save and exit rule editing.

## Alert rule fields

This section describes the fields you fill out to create an alert.

### Alert type

- **Alert name -** Enter a descriptive name. The name will be displayed in the alert rule list, as well as added as `alertname` label to every alert instance that is created from this rule.
- **Alert type -** Select **Cortex / Loki managed alert**.
- **Data source -** Select a Prometheus or Loki data source. Only Prometheus data sources that support Cortex ruler API will be available.
- **Namespace -** Select an existing rule namespace or click **Add new** to create a new one.
- **Group -** Select an existing group within the selected namespace or click **Add new** to create a new one. Newly created rules will be added to the end of the rule group.

![Alert type section screenshot](/img/docs/alerting/unified/rule-edit-cortex-alert-type-8-0.png 'Alert type section screenshot')

### Query

Enter a PromQL or LogQL expression. Rule will fire if evaluation result has at least one series with value > 0. An alert will be created per each such series.

![Query section](/img/docs/alerting/unified/rule-edit-cortex-query-8-0.png 'Query section screenshot')

### Conditions

- **For -** For how long the selected condition should violated before an alert enters `Firing` state. When condition threshold is violated for the first time, an alert becomes `Pending`. If the **for** time elapses and the condition is still violated, it becomes `Firing`. Else it reverts back to `Normal`.

![Conditions section](/img/docs/alerting/unified/rule-edit-cortex-conditions-8-0.png 'Conditions section screenshot')

### Details

Annotations and labels can be optionally added in the details section.

#### Annotations

Annotations are key and value pairs that provide additional meta information about the alert, for example description, summary, runbook URL. They are displayed in rule and alert details in the UI and can be used in contact type message templates. Annotations can also be templated, for example `Instance {{ $labels.instance }} down` will have the evaluated `instance` label value added for every alert this rule produces.

#### Labels

Labels are key value pairs that categorize or identify an alert. Labels are used to match alerts in silences or match and groups alerts in notification policies. Labels are also shown in rule or alert details in the UI and can be used in contact type message templates. For example, it is common to add a `severity` label and then configure a separate notification policy for each severity. Or one could add a `team` label and configure team specific notification policies, or silence all alerts for a particular team.

![Details section](/img/docs/alerting/unified/rule-edit-details-8-0.png 'Details section screenshot')

## Preview alerts

To evaluate the rule and see what alerts it would produce, click **Preview alerts**. It will display a list of alerts with state and value of for each one.
108 changes: 108 additions & 0 deletions docs/sources/alerting/unified-alerting/create-grafana-managed-rule.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
+++
title = "Create Grafana managed alert rule"
description = "Create Grafana managed alert rule"
keywords = ["grafana", "alerting", "guide", "rules", "create"]
weight = 400
+++

# Create a Grafana managed alerting rule

Grafana allows you to create alerting rules that query one or more data sources, reduce or transform the results and compare them to each other or to fix thresholds. These rules will be executed and notifications sent by Grafana itself.

## Add or edit a Grafana managed alerting rule

1. In the Grafana menu hover your cursor over the Alerting (bell) icon.
1. To create a new alert rule, click **New alert rule**. To edit an existing rule, expand one of the rules in the **Grafana** section and click **Edit**.
1. Click on the **Alert type** drop down and select **Grafana managed alert**.
1. Fill out the rest of the fields. Descriptions are listed below in [Alert rule fields](#alert-rule-fields).
1. When you have finished writing your rule, click **Save** in the upper right corner to save the rule,, or **Save and exit** to save and exit rule editing.

## Alert rule fields

This section describes the fields you fill out to create an alert.

### Alert type

- **Alert name -** Enter a descriptive name. The name will be displayed in the alert rule list, as well as added as `alertname` label to every alert instance that is created from this rule.
- **Alert type -** Select **Grafana managed alert**.
- **Folder -** Select a folder this alert rule will belong to. To create a new folder, click on the drop down and type in a new folder name.

![Alert type section screenshot](/img/docs/alerting/unified/rule-edit-grafana-alert-type-8-0.png 'Alert type section screenshot')

### Query

Add one or more [queries]({{< relref "../../panels/queries.md" >}}) or [expressions]({{< relref "../../panels/expressions.md" >}}). You can use classic condition expression to create a rule that will trigger a single alert if it's threshold is met, or use reduce and math expressions to create a multi dimensional alert rule that can trigger multiple alerts, one per matching series in the query result.

#### Rule with classic condition

You can use classic condition expression to create a rule that will trigger a single alert if it's conditions is met. It works about the same way as dashboard alerts in previous versions of Grafana.

1. Add one or more queries
1. Add an expression. Click on **Operation** dropdown and select **Classic condition**.
1. Add one or more conditions. For each condition you can specify operator (`AND` / `OR`), aggregation function, query letter and threshold value.

If a query returns multiple series, then the aggregation function and threshold check will be evaluated for each series.It will not track alert state **per series**. This has implications that are detailed in the scenario below.

- Alert condition with query that returns 2 series: **server1** and **server2**
- **server1** series causes the alert rule to fire and switch to state `Firing`
- Notifications are sent out with message: _load peaking (server1)_
- In a subsequent evaluation of the same alert rule, the **server2** series also causes the alert rule to fire
- No new notifications are sent as the alert rule is already in state `Firing`.

So, as you can see from the above scenario Grafana will not send out notifications when other series cause the alert to fire if the rule already is in state `Firing`. If you want to have alert per series, create a multi dimensional alert rule as described in the section below.

![Query section classic condition](/img/docs/alerting/unified/rule-edit-classic-8-0.png 'Query section classic condition screenshot')

#### Multi dimensional rule

You can use reduce and math expressions to create a rule that will create an alert per series returned by the query.

1. Add one or more queries
2. Add a `reduce` expression for each query to aggregate values in the selected time range into a single value. Not needed in case a query returns a single value per series.
3. Add a `math` expressions with the condition for the rule. Not needed in case a query or a reduce expression already returns 0 if rule should not be firing, or > 0 if it should be firing. Some examples: `$B > 70` if it should fire in case value of B query/expression is more than 70. `$B < $C * 100` in case it should fire if value of B is less than value of C multiplied by 100. If queries being compared have multiple series in their results, series from different queries are matched if they have the same labels or one is a subset of the other.

See or [expressions documentation]({{< relref "../../panels/expressions.md" >}}) for in depth explanation of `math` and `reduce` expressions.

![Query section multi dimensional](/img/docs/alerting/unified/rule-edit-multi-8-0.png 'Query section multi dimensional screenshot')

### Conditions

- **Condition -** Select the letter of the query or expression whose result will trigger the alert rule. You will likely want to select either a `classic condition` or a `math` expression.
- **Evaluate every -** How often the rule should be evaluated, executing the defined queries and expressions. Must be no less than 10 seconds and a multiple of 10 seconds. Examples: `1m`, `30s`
- **Evaluate for -** For how long the selected condition should violated before an alert enters `Alerting` state. When condition threshold is violated for the first time, an alert becomes `Pending`. If the **for** time elapses and the condition is still violated, it becomes `Alerting`. Else it reverts back to `Normal`.

#### No Data & Error handling

Toggle **Configure no data and error handling** switch to configure how the rule should handle cases where evaluation results in error or returns no data.

| No Data Option | Description |
| --------------- | ------------------------------------------------------------------------------------------ |
| No Data | Set alert state to `NoData` and rule state to `Normal` |
| Alerting | Set alert rule state to `Alerting` |
| Ok | Set alert rule state to `Normal` |


| Error or timeout option | Description |
| ----------------------- | --------------------------------------------------- |
| Alerting | Set alert rule state to `Alerting` |
| OK | Set alert rule state to `Normal` |

![Conditions section](/img/docs/alerting/unified/rule-edit-grafana-conditions-8-0.png 'Conditions section screenshot')

### Details

Annotations and labels can be optionally added in the details section.

#### Annotations

Annotations are key and value pairs that provide additional meta information about the alert, for example description, summary, runbook URL. They are displayed in rule and alert details in the UI and can be used in contact type message templates. Annotations can also be templated, for example `Instance {{ $labels.instance }} down` will have the evaluated `instance` label value added for every alert this rule produces.

#### Labels

Labels are key value pairs that categorize or identify an alert. Labels are used to match alerts in silences or match and groups alerts in notification policies. Labels are also shown in rule or alert details in the UI and can be used in contact type message templates. For example, it is common to add a `severity` label and then configure a separate notification policy for each severity. Or one could add a `team` label and configure team specific notification policies, or silence all alerts for a particular team.

![Details section](/img/docs/alerting/unified/rule-edit-details-8-0.png 'Details section screenshot')

## Preview alerts

To evaluate the rule and see what alerts it would produce, click **Preview alerts**. It will display a list of alerts with state and value for each one.
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ export function PreviewRule(): React.ReactElement | null {
<div className={styles.container}>
<HorizontalGroup>
<Button type="button" variant="primary" onClick={onPreview}>
Preview your alert
Preview alerts
</Button>
</HorizontalGroup>
<PreviewRuleResult preview={preview} />
Expand Down

0 comments on commit d6037a4

Please sign in to comment.