Disable access to Alertmanager (user, or sending alerts), if it is not running.

[pstibrany]

What this PR does: This PR disables access to user's Alertmanager (by user, or services trying to send alert) until MultitenantAlertmanager has started and synced configuration for each user.

Call to tenant's AM before MultitenantAlertmanager is running may result in loss of configuration for this tenant.

Checklist

• Tests updated
• [na] Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Tests are failing

[jtlisi]

Is this enough? In addition, shouldn't we require a check to the backend before storing the fallback? Othwerwise, we could end up in a similar situation in the future. For instance, once we start sharding Alertmanager tenants if a request somehow gets routed to an Alertmanager without the tenant loaded it will override the config.

[pstibrany]

Is this enough?

I think yes, for now.

In addition, shouldn't we require a check to the backend before storing the fallback? Othwerwise, we could end up in a similar situation in the future. For instance, once we start sharding Alertmanager tenants if a request somehow gets routed to an Alertmanager without the tenant loaded it will override the config.

That's a valid point, and something keep an eye on when reviewing AM sharding. I think we should also reconsider idea of uploading "empty" config to the same path as user-uploaded config. Eg. we could give empty config ".default" suffix.

[gotjosh]

Good catch @pstibrany !

[jtlisi]

LGTM modulo changelog suggestion

[pracucci]

LGTM!