Alertmanager: AM can deactivate manager for user using fallback config

[pstibrany]

If tenant is assigned "fallback" configuration at the same time as sync of configurations is in progress, tenant's AM may get deactivated (Paused). Consider following scenario:

1. Sync of configuration starts. This consists of a) listing configs, b) loading them to memory, c) and then applying.
2. At the same time as sync is in progress, some tenant is assigned default configuration.

If 2. happens after listing (1a) step, then active AM is created for tenant, but then during (1c) step it will be deactivated, because current sync haven't seen the config file, but can already see active AM. When AM is deactivated, cached in-memory configuration is removed for user (

cortex/pkg/alertmanager/multitenant.go

Line 350 in e2f984c

delete(am.cfgs, user)

).

Such AM will stay deactivated also during the subsequent syncs, because check on line 436 is not triggered. Loaded "cfg.RawConfig" will be empty string, but am.cfgs[cfg.User].RawConfig for missing user (see deletion above) also results in empty string (

cortex/pkg/alertmanager/multitenant.go

Line 436 in e2f984c

} else if am.cfgs[cfg.User].RawConfig != cfg.RawConfig || hasTemplateChanges {

)

End result is that unlucky user will have its AM deactivated (until new configuration is uploaded), and ruler will complain about being unable to push alerts to such user.

/cc @gotjosh

[gotjosh]

There's another case to take into account here: we could end up overwriting the user's configuration.

Two scenarios are possible (assuming a fallback configuration is specified):

1. Receive an alert request before the initial sync on startup. If we received an alert before the initial sync is completed, we could end up uploading a fallback configuration because we assume there's no alertmanager running and end up deleting the user-defined configuration.

2. User does not have an alertmanager, uploads a configuration (which has not been synced yet) and receives an alert will end up overriding the previously uploaded configuration.

While #1 is technically not possible after #3664 when in sharding mode (we wait for the alertmanager to finish doing aninitial sync, before manually transitioning it to active) #2 is still possible under the current scheme.

My suggestion would be to upload the "fallback" configuration into a different object (as a different name or it could also be a marker telling it to use the fallback configuration we already have cached in-memory), and then have the configuration sync process look at either the fallback or user-defined configuration within the object storage.