Alert severity not correctly set on PagerDuty Alert

[dbonatto]

I'm trying to migrate to the PagerDuty API v2. I already changed to use the routing_key in the configuration so that AM would use the API v2. What I can not find how to do is passing on to PagerDuty the severity that is declared as a label in the alert.

curl -XPOST -k https://somehost/api/v1/alerts -d '[{
        "status": "firing",
        "labels": {
                "alertname": "TestAlertWarning-13127",
                "service": "my-service",
                "team": "C Team",
                "severity":"warning",
                "instance": "TestAlertWarning-13127.example.net"
        },
        "annotations": {
                "summary": "High latency is high!"
        },
        "generatorURL": "http://prometheus.int.example.net/<generating_expression>"
}]'

What is happening now is that AM is sending a "error" severity to PagerDuty, which looking in the code I could see that is the default behavior when no severity is set.

Not sure if it is a code or documentation issue, would appreciate some help.

• Alertmanager version:

Branch: HEAD
BuildDate: 20180112-10:32:46
BuildUser: root@d83981af1d3d
GoVersion: go1.9.2
Revision: fb713f6
Version: 0.13.0

• Prometheus version:

Version 1.8.2
Revision 5211b96d4d1291c3dd1a569f711d3b301b635ecb
Branch HEAD
BuildUser root@1412e937e4ad
BuildDate 20171104-16:09:14
GoVersion go1.9.2

• Alertmanager configuration file:

  - receiver: CTeam-pagerduty-alert
    match:
      team: C Team
...
- name: CTeam-pagerduty-alert
  pagerduty_configs:
  - send_resolved: true
    routing_key: <secret>
    url: https://events.pagerduty.com/v2/enqueue
    client: '{{ template "pagerduty.default.client" . }}'
    client_url: '{{ template "pagerduty.default.clientURL" . }}'
    description: '{{ template "pagerduty.default.description" .}}'
    details:
      firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'

[stuartnelson3]

Severity is set in pagerduty_configs:

- name: CTeam-pagerduty-alert
  pagerduty_configs:
  - send_resolved: true
    routing_key: <secret>
    severity: warning

[dbonatto]

Sorry, this was one of the tests I've done.

If you remove it, it will have the behavior I described. Its not passing on the severity and I can't find in the documentation how to do it.

[stuartnelson3]

Ah, I misunderstood, sorry about that. As it is now, the severity can only be defined statically in the receiver.

To pass on the severity in the label, the code needs to be slightly updated here:

alertmanager/notify/impl.go

Line 509 in 6a3dfaf

Severity: n.conf.Severity,

n.conf.Severity needs to become tmpl(n.conf.Severity). Then you can use the normal templating to access the common severity label.

I would recommend against this, as if an alert group has mixed severity labels (one is severe, one is warning), then that common label isn't there for templating. I'm not sure how you have your grouping configured, though.

I would recommend in your routes defining a match on severity=warning and a separate severity=critical (or whatever the levels are), and having them point to two different receivers. It's more verbose, but perhaps clearer to others reading the config that don't know how the grouping and common labels work.

That having been said, if you want to add templating to the severity in the link above, that would be fine by me (although I would still recommend being explicit).

[dbonatto]

Thanks for reopening.

So in my use of AM, we have around 10 teams and many more services. Each team will decide on the severity of an alert based on the type of problem and the environment in which the problem is happening. Also, PagerDuty makes use of the Severity to define what is the urgency of a alert and with that decide how to contact on call engineers.

So in my opinion it would be very useful to have a way to pass the severity configured in the alert directly to PagerDuty, even if that is not the default behavior. Looks like that is the alternative you suggested and I'm very much in favor of it.

Creating routing rules for all teams I have and for all four valid seventies in PagerDuty (Info, Warning, Error and Critical) will be a little bit messy. Still, looks like that is the alternative I have for now.

Also, I think offering a template for other fields provided by the PagerDuty API v2, such as Group and Component might be useful, so we can have a more generic configuration file and have information being dynamically set according to what is set in the alert.

[dbonatto]

@stuartnelson3 I went ahead and created the PR with the solution you mentioned in case you agree with integrating it.

Also, the routing solution works, still it creates a giant config file. Every team will need to have something like this:

  - match:
      team: C Team
    routes:
    - receiver: CTeam-pagerduty-critical
      match:
        severity: critical
    - receiver: CTeam-pagerduty-warning
      match:
        severity: warning
    - receiver: CTeam-pagerduty-info
      match:
        severity: info
    - receiver: CTeam-pagerduty-error
      match:
        severity: error

In the end, more templates would make the configuration simpler.

[dbonatto]

@stuartnelson3 Please let me know if this is what you are looking for: prometheus/docs#956