Allow alerting to be configured on a per canary basis

[stefanprodan]

Currently alerting via Slack/MS Teams can be configured globally with Flagger command flags. This approach has several limitations as it's not possible to specify different channels (ref: #128 #242) or configure the verbosity (ref: #427) on a per canary basis.

To make the alerting move flexible, the canary analysis could be extended with a list of alerts that would reference an alert provider, for each alert users can configure the severity level. The alerts section would override the global setting.

Alert provider specifications

Kubernetes custom resource:

type AlertProviderSpec struct {
	// Type of provider
	Type string `json:"type"`

	// Alert channel for this provider
	// +optional
	Channel string `json:"channel,omitempty"`

	// Bot username for this provider
	// +optional
	Username string `json:"username,omitempty"`
	
	// HTTP(S) webhook address of this provider
	// +optional
	Address string `json:"address,omitempty"`

	// Secret reference containing the provider webhook URL
	// +optional
	SecretRef *corev1.LocalObjectReference `json:"secretRef,omitempty"`
}

The alert provider type can be: slack, msteams, rocket or discord. When set to discord, Flagger will use Slack formatting and will append /slack to the Discord address.

When not specified, channel defaults to general and username defaults to flagger.

When secretRef is specified, the Kubernetes secret must contain a data field named address, the address in the secret will take precedence over the address field in the provider spec.

Slack example:

apiVersion: flagger.app/v1beta1
kind: AlertProvider
metadata:
  name: dev-slack
  namespace: flagger
spec:
  type: slack
  channel: dev-alerts
  username: flagger
  # webhook address (ignored if secretRef is specified)
  address: https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK
  # secret containing the webhook address (optional)
  secretRef:
    name: dev-slack-url
---
apiVersion: v1
kind: Secret
metadata:
  name: dev-slack-url
  namespace: flagger
data:
  address: <encoded-url>

Canary analysis specifications

The canary analysis can have a list of alerts, each alert referencing an alert provider:

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: app
  namespace: staging
spec:
  canaryAnalysis:
    alerts:
      - name: "dev team Slack"
        severity: error
        providerRef:
          name: dev-slack
          namespace: flagger
      - name: "qa team Discord"
        severity: warn
        providerRef:
          name: qa-discord
      - name: "on-call MS Teams"
        severity: info
        providerRef:
          name: on-call-msteams

Alert fields:

• name (required)
• severity levels: info, warn, error (default info)
• providerRef.name alert provider name (required)
• providerRef.namespace alert provider namespace (defaults to the canary namespace)

When the severity is set to warn, Flagger will alert when waiting on manual confirmation or if the analysis fails.
When the severity is set to error, Flagger will alert only if the canary analysis fails.

[heubeck]

Hello @stefanprodan,
thank you for your dedication on Flagger which is really a great product.

Are you planning to create some kind of generic webhook AlertProvider (e.g. spec.type: webhook).
We are using the eventWebhook for publishing to our on-premise Rocket.Chat and struggling with securing the apikey as part of the url.

Furthermore (maybe not as part of this issue) it would be great if the event payload contained more information about the canary definition that's processed, e.g. containers image tag.

Thank you and BR

[stefanprodan]

@heubeck I think you can use the Slack alert provider (with secretRef) for Rocket and parse the request body in a Rocket script to transform the Slack fields. The way alerting works is different to the events, with events you get everything that Flagger does, with alerts you get a compact message that summarises what's going on. Do you want to see in Rocket all events or just the alerts?

@stefanprodan thank you - you are right. We are mainly interessted in the events since we can filter or group in the Rocket script. I like the idea of using the Slack messages.

So we will have a try with your example in the helm chart:

#env:
#- name: SLACK_URL
#  valueFrom:
#    secretKeyRef:
#      name: slack
#      key: url

Thank you!

[stefanprodan]

@jug-in open an issue for this, Flagger has to read the event URL from env vars instead of just looking at the command arg, once this is implemented you can use secretKeyRef in the chart.

[heubeck]

Hey @stefanprodan you are really fast ;)

I had a look at the Slack notifications - they contain just begin and end of a canary, right?
The rocket provider you introduced would receive the same messages as I understand.

Your proposal to read the event URL from env sounds very handy.

BTW: when setting env.SLACK_URL in the helm chart, Flagger fails at https://github.com/weaveworks/flagger/blob/0.23.0/pkg/notifier/slack.go#L68 when not providing a dummy slack.url too.

[stefanprodan]

The rocket provider you introduced would receive the same message

Yes, this issue/PR is about alerts not events.

BTW: when setting env.SLACK_URL in the helm chart, Flagger fails

You need to provide -slack-channel for Slack to work.

[heubeck]

You need to provide -slack-channel for Slack to work.

Yes, and also a slack.url in the helm chart since otherwise the -slack-channel is not used:

# deployment.yaml:
          {{- if .Values.slack.url }}
          - -slack-url={{ .Values.slack.url }}
          - -slack-user={{ .Values.slack.user }}
          - -slack-channel={{ .Values.slack.channel }}
          {{- end }}