Slack Integration does not respect rate limiting

[gavk34]

What did you do?
Starting seeing the following errors in AlertManager

level=error ts=2019-11-21T09:02:58.967Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="cancelling notify retry for \"slack\" due to unrecoverable error: unexpected status code 429: {\"retry_after\":1,\"ok\":false,\"error\":\"rate_limited\"}" context_err=null
level=error ts=2019-11-21T09:02:58.967Z caller=dispatch.go:266 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="cancelling notify retry for \"slack\" due to unrecoverable error: unexpected status code 429: {\"retry_after\":1,\"ok\":false,\"error\":\"rate_limited\"}"

What did you expect to see?
I would expect the dispatcher to respect the retry_after and try later rather than failing

What did you see instead? Under which circumstances?
Increase volume of alerts triggered

Environment

• System information:

  Docker container

• Alertmanager version:

  v0.19.0

• Prometheus version:

  N/A

• Alertmanager configuration file:
  N/A - contains confidential information

• Prometheus configuration file:
  N/A

• Logs:

level=error ts=2019-11-21T09:02:58.967Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="cancelling notify retry for \"slack\" due to unrecoverable error: unexpected status code 429: {\"retry_after\":1,\"ok\":false,\"error\":\"rate_limited\"}" context_err=null
level=error ts=2019-11-21T09:02:58.967Z caller=dispatch.go:266 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="cancelling notify retry for \"slack\" due to unrecoverable error: unexpected status code 429: {\"retry_after\":1,\"ok\":false,\"error\":\"rate_limited\"}"

[roidelapluie]

Alertmanager has its internal retry mechanism. Using a value provided by upstream would have bad effects on high availability as you could get multiple times the same notification.

[gavk34]

@roidelapluie If that is the case it might be good to ignore such errors in alertmanager_notifications_failed_total as they are triggering alerts for us, even though as you say AlertManager is able to handle this scenario internally.

[roidelapluie]

Mmh wait. No you are right, alertmanager will not retry 4XX errors.

[roidelapluie]

See #2112

[gavk34]

@roidelapluie @simonpasquier Thanks for the quick fix guys! 👍

[gavk34]

@roidelapluie @simonpasquier Any idea when the next AlertManager release is going to be? I see Prometheus got updated this month, but AlertManager hasn't seen an update since September. Thanks

[simonpasquier]

I'm planning a new release soon (eg this week probably). As you noticed, it's been almost 3 months since the last one.

[simonpasquier]

https://github.com/prometheus/alertmanager/releases/tag/v0.20.0-rc.0 is ready to test 😃

[vvxxvvxx]

encounter the similar issue using vmalertmanager v0.24.0 #3189