Overview

A lightweight Go Web Server that receives POST alert messages from Prometheus Alertmanager and sends it to a Cisco Webex Teams Room using the Webex API.

Synopsis

Alertmanager doesn't support sending to Cisco Webex Teams out of the box. Fortunately, they allow you to use a generic webhook_config for cases like this. This project was inspired from prometheus-msteams's which was written in Go for Microsoft Teams.

Table of Contents

• Getting Started (Quickstart)
  • Installation
  • Setting up Prometheus Alert Manager
  • Simulating a Prometheus Alerts to Teams Channel
• Sending Alerts to Multiple Teams Channel
  • Creating the Configuration File
  • Setting up Prometheus Alert Manager
• Customise Messages to MS Teams
  • Customise Messages per MS Teams Channel
  • Use Template functions to improve your templates
• Configuration
• Kubernetes Deployment
• Contributing

Getting Started (Quickstart)

How it works.

Installation

We always recommend to use the latest stable release!

OPTION 1: Run using docker.

docker run -d -p 2000:2000 \
    --name="promteams" \
    -e TEAMS_ACCESS_TOKEN="NzhiODhlZDYtZTF..." \
    -e TEAMS_ROOM_ID="Y2lzY29zcGFyazovL3VzL1JPT00vYWZmMTllNTAtY..." \
    infonova/prometheus-webexteams

OPTION 2: Run using binary.

Download the binary for your platform and the default card template from RELEASES, then run the binary in the same directory as you have stored the default-message-card.tmpl like the following:

./bin/prometheus-webexteams-<goos>-<goarch> -teams-access-tocken "NzhiODhlZDYtZTF..." \
  -teams-room-id "Y2lzY29zcGFyazovL3VzL1JPT00vYWZmMTllNTAtY..."

OPTION 3: If you are going to deploy this in a Kubernetes cluster, checkout the Kubernetes Deployment Guide.

Setting up Prometheus Alertmanager

By default, prometheus-webexteams creates a request uri handler /alertmanager.

route:
  group_by: ['alertname']
  group_interval: 30s
  repeat_interval: 30s
  group_wait: 30s
  receiver: 'prometheus-webexteams'

receivers:
- name: 'prometheus-webexteams'
  webhook_configs: # https://prometheus.io/docs/alerting/configuration/#webhook_config 
  - send_resolved: true
    url: 'http://prometheus-webexteams:2000/alertmanager' # the prometheus-webexteams proxy

If you don't have Prometheus running yet and you wan't to try how this works,
try stefanprodan's Prometheus in Docker to help you install a local Prometheus setup quickly in a single machine.

Simulating a Prometheus Alerts to Teams Room

Create the following json data as prom-alert.json.

{
    "version": "4",
    "groupKey": "{}:{alertname=\"high_memory_load\"}",
    "status": "firing",
    "receiver": "teams_proxy",
    "groupLabels": {
        "alertname": "high_memory_load"
    },
    "commonLabels": {
        "alertname": "high_memory_load",
        "monitor": "master",
        "severity": "warning"
    },
    "commonAnnotations": {
        "summary": "Server High Memory usage"
    },
    "externalURL": "http://docker.for.mac.host.internal:9093",
    "alerts": [
        {
            "labels": {
                "alertname": "high_memory_load",
                "instance": "10.80.40.11:9100",
                "job": "docker_nodes",
                "monitor": "master",
                "severity": "warning"
            },
            "annotations": {
                "description": "10.80.40.11 reported high memory usage with 23.28%.",
                "summary": "Server High Memory usage"
            },
            "startsAt": "2018-03-07T06:33:21.873077559-05:00",
            "endsAt": "0001-01-01T00:00:00Z"
        }
    ]
}

curl -X POST -d @pkg/card/testdata/prometheus_fire_request.json http://<hostname|ip>:2000/alertmanager

The teams room should received a message.

Sending Alerts to Multiple Teams Rooms

You can configure this application to serve 2 or more request path and each path can use a unique Teams room to post.

This can be achieved by supplying the application a configuration file.

Creating the Configuration File

Create a yaml file with the following format.

connectors:
  - request_path: high-prio-ch
    access_token: NzhiODhlZDYtZ...
    room_id: Y2lzY29zcGFyazovL...
    template_file: ./resources/default-message-card.tmpl
    webhook_url: https://webexapis.com/v1/messages
    escape_underscores: false
- request_path: low-prio-ch
    access_token: NzhiODhlZDYtZ...
    room_id: Y2lzY29zcGFyazovL...
    template_file: ./resources/default-message-card.tmpl
    webhook_url: https://webexapis.com/v1/messages
    escape_underscores: false

When running as a docker container, mount the config file in the container and set the CONFIG_FILE environment variable.

docker run -d -p 2000:2000 \
    --name="promteams" \
    -v /tmp/config.yml:/tmp/config.yml \
    -e CONFIG_FILE="/tmp/config.yml" \
    infonova/prometheus-webexteams

When running as a binary, use the -config-file flag.

./bin/prometheus-webexteams-<goos>-<goarch> -config-file /tmp/config.yml

This will create the request uri handlers /high-prio-ch and /low-prio-ch.

To validate your configuration, see the /config endpoint of the application.

curl localhost:2000/config

[
  {
    "RequestPath": "high-prio-ch",
    "AccessToken": "NzhiODhlZDYtZ...",
    "RoomId": "Y2lzY29zcGFyazovL...",
    "TemplateFile": "./resources/default-message-card.tmpl",
    "WebhookURL": "https://webexapis.com/v1/messages",
    "EscapeUnderscores": false
  },
  {
    "RequestPath": "low-prio-ch",
    "AccessToken": "NzhiODhlZDYtZ...",
    "RoomId": "Y2lzY29zcGFyazovL...",
    "TemplateFile": "./resources/default-message-card.tmpl",
    "WebhookURL": "https://webexapis.com/v1/messages",
    "EscapeUnderscores": false
  }
]

Setting up Prometheus Alertmanager

Considering the prometheus-webexteams config file settings, your Alert Manager would have a configuration like the following.

route:
  ...
  routes:
    - receiver: high_prio_receiver
      match:
        severity: critical
    - receiver: low_prio_receiver
      match:
        severity: warning

receivers:
- name: 'high_prio_receiver'
  webhook_configs:
    - send_resolved: true
      url: 'http://<servicename>:2000/high_prio_ch' # request handler 1
- name: 'low_prio_receiver'
  webhook_configs:
    - send_resolved: true
      url: 'http://<servicename>:2000/low_prio_ch' # request handler 2

Use Template functions to improve your templates

You can use

• all of the existing sprig template functions except the OS functions env and expandenv
• some well known functions from Helm: toToml, toYaml, fromYaml, toJson, fromJson

Configuration

All configuration from flags can be overwritten using environment variables.

E.g, -config-file is CONFIG_FILE, -debug is DEBUG, -log-format is LOG_FORMAT.

Usage of prometheus-webexteams:
  -config-file string
        The connectors configuration file.
  -debug
        Set log level to debug mode. (default true)
  -escape-underscores
        Automatically replace all '_' with '\_' from texts in the alert.
  -http-addr string
        HTTP listen address. (default ":2000")
  -idle-conn-timeout duration
        The HTTP client idle connection timeout duration. (default 1m30s)
  -jaeger-agent string
        Jaeger agent endpoint (default "localhost:6831")
  -jaeger-trace
        Send traces to Jaeger.
  -log-format string
        json|fmt (default "json")
  -max-idle-conns int
        The HTTP client maximum number of idle connections (default 100)
  -request-uri string
        The default request URI path where Prometheus will post to. (default "alertmanager")
  -teams-access-token string
        The access token to authorize the requests.
  -teams-room-id string
        The room specifies the target room of the messages.
  -teams-webhook-url string
        The default Webex Teams webhook connector. (default "https://webexapis.com/v1/messages")
  -template-file string
        The default Webex Teams Message Card template file. (default "resources/default-message-card.tmpl")
  -tls-handshake-timeout duration
        The HTTP client TLS handshake timeout. (default 30s)
  -version
        Print the version

Kubernetes Deployment

See Helm Guide.