Static alert channel length of 200 is a bottleneck in Alertmanager

[siavashs]

Alertmanager memory provider has a static channel length of 200:

alertmanager/provider/mem/mem.go

Line 30 in f6b942c

const alertChannelLength = 200

This value has been there with no description for almost a decade since introduction in cc5aab6

It seems like a bottleneck in Alertmanager, since a slow alert pipeline can result in API timeouts on POST /api/v2/alerts as the pipeline struggles to consume from the channel.
Root cause of a slow pipeline can be from any of:

• high volume of alerts
• high volume of suppressed alerts (inhibition or silences), regex matching can have significant affect here
• slow receivers

With #4364 the channel length becomes configurable, allowing bigger alert spikes to be queued for processing by Alertmanager.

[grobinson-grafana]

I believe the notification pipeline (flushing) is independent to the dispatcher subscribing to alerts from the API as these happen in seperate goroutines. Is it possible the bottleneck is somewhere else?

[siavashs]

I believe the notification pipeline (flushing) is independent to the dispatcher subscribing to alerts from the API as these happen in seperate goroutines. Is it possible the bottleneck is somewhere else?

Regarding dispacher -> notification, I have to double-check.

But Inhibitor and Dispatcher could be slow themselves. This is what I understood by going through the code:

The shared alerts memory provider (calling it AMP moving forward) is created here:

alertmanager/cmd/alertmanager/main.go

Line 345 in f6b942c

alerts, err := mem.NewAlerts(context.Background(), marker, *alertGCInterval, nil, logger, prometheus.DefaultRegisterer)

API

AMP is passed to the API here:

alertmanager/cmd/alertmanager/main.go

Lines 369 to 370 in f6b942c

	api, err := api.New(api.Options{
	Alerts: alerts,

API puts alerts into the AMP:

alertmanager/api/v2/api.go

Line 362 in f6b942c

if err := api.alerts.Put(validAlerts...); err != nil {

Inhibitor

AMP is passed to the inhibitor here:

alertmanager/cmd/alertmanager/main.go

Line 464 in f6b942c

inhibitor = inhibit.NewInhibitor(alerts, conf.InhibitRules, marker, logger)

Inhibitor subscribes to the AMP here:

alertmanager/inhibit/inhibit.go

Line 60 in f6b942c

it := ih.alerts.Subscribe()

Dispatcher

AMP is passed to the dispatcher here:

alertmanager/cmd/alertmanager/main.go

Line 495 in f6b942c

disp = dispatch.NewDispatcher(alerts, routes, pipeline, marker, timeoutFunc, nil, logger, dispMetrics)

Dispatcher subscribes to the AMP here:

alertmanager/dispatch/dispatch.go

Line 144 in f6b942c

d.run(d.alerts.Subscribe())

Memory Provider

AMP writes the alert to each listener channel (each has capacity of 200):

alertmanager/provider/mem/mem.go

Lines 252 to 257 in f6b942c

	for _, l := range a.listeners {
	select {
	case l.alerts <- alert:
	case <-l.done:
	}
	}

So while the Inhibitor and Dispatcher get their own alert channels with a size of 200, API writes will be blocked if either are slow enough to consume from their dedicated channels.

[grobinson-grafana]

Yes I agree! What I'm concerned about is increasing the size of the channel doesn't fix the fundamental problem, and neither the dispatcher or inhibitor should be so slow that the channel fills up. Do you have any profiles you can share so we can better understand what the root cause is?