Alertmanager cluster peers have spiky routines

[siavashs]

We have observed that the Alertmanager cluster peer on position 0 (which is the first to send notifications) has almost flat routines metric.
While the other 2 peers in following positions have more spiky routines up to 3K routines (+30%).
In the below graph we can see how the yellow instance at position zero had a flat routines graph but when the positions are swapped by renaming peers, the blue is at position zero. The switch in positions moves the spiky routines symptom between the peers.

So the fanout stage creates all these go routines here

alertmanager/notify/notify.go

Lines 491 to 514 in edb9a4d

	// Exec attempts to execute all stages concurrently and discards the results.
	// It returns its input alerts and a types.MultiError if one or more stages fail.
	func (fs FanoutStage) Exec(ctx context.Context, l *slog.Logger, alerts ...*types.Alert) (context.Context, []*types.Alert, error) {
	var (
	wg sync.WaitGroup
	me types.MultiError
	)
	wg.Add(len(fs))
	for _, s := range fs {
	go func(s Stage) {
	if _, _, err := s.Exec(ctx, l, alerts...); err != nil {
	me.Add(err)
	}
	wg.Done()
	}(s)
	}
	wg.Wait()
	if me.Len() > 0 {
	return ctx, alerts, &me
	}
	return ctx, alerts, nil
	}

and then the wait stage will wait peer timeout * position here

alertmanager/notify/notify.go

Lines 599 to 607 in edb9a4d

	// Exec implements the Stage interface.
	func (ws *WaitStage) Exec(ctx context.Context, _ *slog.Logger, alerts ...*types.Alert) (context.Context, []*types.Alert, error) {
	select {
	case <-time.After(ws.wait()):
	case <-ctx.Done():
	return ctx, nil, ctx.Err()
	}
	return ctx, alerts, nil
	}

This seems very unoptimised and wasteful, we could potentially redesign these stages to wait before creating the routines instead of creating, waiting and then destroying.

[grobinson-grafana]

I'm not sure non-position zero peers are creating more goroutines, it's just that because of the WaitStage, these goroutines live long enough to be scraped in the go_goroutines gauge. Each peer should create the exact same number of goroutines.

[siavashs]

I think we don't need to create this and can schedule a function call instead of running and waiting.
Similar to the fix done to AGs in #4632

[grobinson-grafana]

I'm not sure I understand what problem the PR solves. It looks like it replaces long lived goroutines with short-lived go routines via timer.AfterFunc, but the tradeoff here is much harder to reason about control flow. What is the problem being solved?

[siavashs]

The PR fixes the issue described in #4503
While spawning go routines and waiting for a timer is OK, the current AG creation logic can result in unlimited routines.
In a scenario when one use a config like group_by: [instance], Alertmanager creates AGs = cardinality of instance label and this can become a very spiky or heavy workload for both maintenance and garbage collector.
For example in Cloudflare's setup we have observed spikes of 10s of thousands of routines and even 100k routines in few cases.

[grobinson-grafana]

Sorry I didn't understand the issue described in #4503 either. You say that we create one goroutine per aggregation group, which is entirely correct, but I'm not understanding why that is a problem? Goroutines are meant to be extremely cheap for the runtime in terms of their stack size and in terms of scheduling.

I'm also not sure I follow about the garbage collection penalty. You want to swap cardinality of instance label goroutines with number of flushes x cardinality of instance label. Instead of each aggregation group having one long lived goroutine, they create a new one every flush which then needs to be garbage collected? This will create significantly more goroutines over time than the implementation we have today, so if garbage collection pressure is an issue, this change should make it worse, not better?

[siavashs]

Go routines are cheap but not free, and they are also not created once. The current setup creates and destroys too many routines in both dispatch and notify. This is not a problem at smaller scale but we suspect it is a problem at our scale since we see a lot of GC activity.

Anyway, it seems a safe change to test in our production canary and it needs to run for a while so we can compare it agains unpatched versions.

[grobinson-grafana]

A single goroutine is created once at the start of the aggregation group and then that goroutine lives for the entire lifetime of the aggregation group? I'm struggling to understand how that is more expensive than creating a new goroutine on every flush?

[siavashs]

Dispatch maintenance stops the routines and deletes the groups, so they are not really persisted forever.
This also happens on every config reload.

[grobinson-grafana]

Dispatch maintenance does do that but for empty groups only. An aggregation group is considered to have reached the end of its lifetime when all alerts in it have resolved.