Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

config/notifiers: PagerDuty details override defaults #2495

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

BenoitKnecht
Copy link
Contributor

According to the Alertmanager documentation, the details option of
pagerduty_config defaults to

[ details: { <string>: <tmpl_string>, ... } | default = {
  firing:       '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
  resolved:     '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
  num_firing:   '{{ .Alerts.Firing | len }}'
  num_resolved: '{{ .Alerts.Resolved | len }}'
} ]

Which suggests that if details is provided in the configuration, the entire
option is overridden.

But instead, the individual keys of details are being set to the values above
if the corresponding key isn't present in the configuration. This means it's
impossible to remove those keys entirely; setting them to an empty string would
still have them displayed in PagerDuty.

This commit implements the documented behavior, i.e. it sets
PagerDutyConfig.Details to DefaultPagerdutyDetails only if the former is
nil.

@BenoitKnecht
Copy link
Contributor Author

While I think this corresponds to the documented behavior, I recognize that it would break configuration that relied on the previous implementation.

If you prefer, I can propose a patch that keeps the old behavior but removes the details keys whose values are empty strings, so that it become possible to undefine those default keys.

According to the Alertmanager documentation, the `details` option of
`pagerduty_config` defaults to

```
[ details: { <string>: <tmpl_string>, ... } | default = {
  firing:       '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
  resolved:     '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
  num_firing:   '{{ .Alerts.Firing | len }}'
  num_resolved: '{{ .Alerts.Resolved | len }}'
} ]
```

Which suggests that if `details` is provided in the configuration, the entire
option is overridden.

But instead, the individual keys of `details` are being set to the values above
if the corresponding key isn't present in the configuration. This means it's
impossible to remove those keys entirely; setting them to an empty string would
still have them displayed in PagerDuty.

This commit implements the documented behavior, i.e. it sets
`PagerDutyConfig.Details` to `DefaultPagerdutyDetails` only if the former is
`nil`.

Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>
@BenoitKnecht BenoitKnecht force-pushed the pagerduty-default-details branch from 22ed3be to 2661162 Compare March 1, 2021 17:14
@levshvarts
Copy link

While I think this corresponds to the documented behavior, I recognize that it would break configuration that relied on the previous implementation.

If you prefer, I can propose a patch that keeps the old behavior but removes the details keys whose values are empty strings, so that it become possible to undefine those default keys.

Seems like, from the backwards compatibility point of view, this is the way to go. And even though having an empty string hold a special significance is not super clean, this would not go against any other types of default values in prometheus/alertmanager.

levshvarts added a commit to mobiledgex/edge-cloud-infra that referenced this pull request Apr 20, 2021
Reformatted pagerduty notification to use default details labels.
There is an issue being tracked on alertmanager - prometheus/alertmanager#2495, which we'll need in order to clean this up a bit more, but for now we will just reuse the existing default labels for pagerduty details.
@stale stale bot added the stale label Jun 18, 2021
@Preisschild
Copy link

Just ran into the same unexpected behavior. Is there a reason why this wasn't reviewed?

@Aaron-ML
Copy link

@levshvarts @BenoitKnecht Any chance we can get some eyes on this? Running into this myself and I was able to blank them out in my template however pagerduty stops ordering these fields after exceeding 32 fields total, including blank fields.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants