Add generic pluggable exec notifier

[UiP9AV6Y]

A generic, pluggable notifier would reduce the number of feature requests for new notifier implementations.

Motivation

The current recommendation for integrating services ($SVC) into Prometheus Alertmanager (AM) is to use the webhook notifier. If the $SVC does not support AM natively, a bridge has to be used in order to translate the webhook DTO into whatever data packets the receiving end supports. When integrating AM into many such services, this results in many additional services being brought up, all of which need to be scaled, placed, monitored, ... . While i am not against microservices per-se, deploying AM-bridges seems like an overkill to me, given that these services "only" translate. While i could write a generic bridge that supports multiple target services, i was thinking about a pluggable solution for AM.

Providing a generic notifier implementation based on os/exec would allow operators to bring their own implementations without having to deal with scaling/monitoring/...

Proposal

Notifier implementation based on os/exec which runs a command for each notification request. Data (similar to the webhook DTO) is provided via STDIN. The exit code is used to signal the result (0=success, 3=retry, else=failure)

I do have a PoC that i can clean up and provide as first draft via MR.

[TheMeier]

I like the proposal. But has has been discussed multiple times before, allowing optional templating in the webhook notifier would be practical, simple solution for most targets. I would argue it could replace most of the specific notifiers.

[UiP9AV6Y]

I like the proposal. But has has been discussed multiple times before, allowing optional templating in the webhook notifier would be practical, simple solution for most targets. I would argue it could replace most of the specific notifiers.

the webhook notifier is HTTP-based though, so targets with other protocols (SNMP, IRC, ...) still require a bridge for integration.

templating the webhook request body will also only cover basic integrations. some of the existing notifier implementations perform additional logic apart from DTO transformation (e.g. HTTP headers with request signatures in the SNS notifier, preflight validation in the Jira notifier)

[SuperQ]

While I understand the desire, I don't think we should do this as a matter of policy.