Arbitrary notifier integrations with Jsonnet

[fabxc]

Similar to Prometheus SD integrations, Alertmanager struggles with an increasing demand for more notifier integrations. We can only maintain so many of them ourselves, contributed integrations are often not great in quality and/or have no long-term maintainers.

Many requested integrations are not designed for reliable alert delivery, yet there's an understandable demand by people with low-risk environments (personal websites etc.). As Alertmanager really is only about making single API calls, which are generally REST APIs using JSON, it's not too crazy think about templating such calls.

Jsonnet is a JSON data templating language by Google. In the past there was no Go implementation but now there's one open sourced.

From my understanding it has reasonably matured semantics (certainly better than us providing something on top of Go templating).
I'd like to consider it to allow people templating of JSON payloads with it. Combined with templating of request URLs, this should be sufficient to let people integrate with almost anything today.

I realize its throwing another templating language into the game and even nesting them if some JSON fields should be templated additionally.
But generally those would be write-once per integration and are easy to share among users without us having any maintenance burden. Ultimately, we could probably even implement a lot of the existing integrations with it.

I'd like to investigate this. Would definitely need a proposal doc though.

@stuartnelson3 @leominov @brian-brazil

[brian-brazil]

One thing to consider is that errors are very inconsistently signalled. There's a good mix of 4xx, 5xx and message content - sometimes within the same integration.

[xMTinkerer]

It seems that accepting pull requests for new notifiers is no longer happening as the burden of maintaining them rests finally on the maintainers of alertmanager. In addition to the Telegram and Jira notifiers, AmazonSNS was not accepted (#709).
Having some kind of plugin system for these will help alleviate the burden off the alertmanager maintainers and back onto the plugin maintainer where it belongs. There will be an ever growing list of new notifiers or snowflake things users want so I would encourage moving that burden to those who can and want to have it.

[stuartnelson3]

https://github.com/hashicorp/go-plugin was recently brought to my attention, which seems like a well-used solution

@gouthamve @brancz might be another issue for GSoC?

[brian-brazil]

How would that differ from the existing webhook? It looks like a slightly different way of doing the same thing.

[stuartnelson3]

It adds a degree of type safety since a plugin has to implement the interface we describe (in this case, the Notifier interface). I would initially propose using the basic rpc implementation, but defining the interface via gRPC would allow plugins in any language.

The communication is done via a unix socket within the system, which is managed by the library. Operationally I think this could be simpler than coordinating between AM config and the webhook's location. Deploying a single docker container packaged with the correct plugins could also be a win vs. deploying AM and various webhooks (in a kubernetes-like environment).

The bi-directional communication could (hopefully) make the connection between AM and a plugin feel like writing directly for the codebase.

Tangentially, success with this might lead to implementing a similar plugin SD architecture in prometheus (no idea if that has been mentioned at all).

[brian-brazil]

That sounds like a more complicated and Go-specific way of doing things. There's nothing stopping users deploying all their webhooks into one binary today, the API/information a notifier gets is the same as what the webhook uses (i.e. there is no bi-directional communication), and Go plugins don't seem viable in our use case as we basically would have to distribute them all.

From Prometheus SD the plan is to stick with file_sd, but there'll be an example Go program that takes the internal SD API and writes out the file_sd file. A similar thing could work here, though given the simpler use case I'm not sure it'd save much effort.