Enable support for http proxy for alertmanager receivers

[Fantaztig]

Describe the bug

Currently it's not possible to configure a proxy for alertmanager receivers.
This is pretty annoying since the receivers also ignore a proxy set via HTTPS_PROXY env variables.

To Reproduce

• Set up mimir with alertmanager
• Try to configure a webhook with http proxy via the grafana UI

Expected behavior

It should be possible to specify the proxy either when configuring the contact point/receiver or globally via environment variables.

Additional Context

It seems this was fixed recently in cortex as well, see: cortexproject/cortex#4741

Setting HTTP(S)_PROXY env variables is a bad solution for this imho as the proxy is currently also used for replication between alertmanager instances in an HA deployment.

[replay]

I think this makes sense, would you be interested in creating a PR based on the solution in Cortex?

[supergicko]

pr on the way

#2050

br

[pracucci]

I prefer to not allow it by default, but I'm open to add an advanced configuration option to enable it (and mention it in the doc).

[supergicko]

idk what "advanced configuration option" would mean in this context :/

[pracucci]

idk what "advanced configuration option" would mean in this context :/

In Mimir, each configuration option is classified either as basic, advanced, experimental or deprecated. See:
https://grafana.com/docs/mimir/next/operators-guide/configuring/reference-configuration-parameters/#parameter-lifecycle

The classification is very simple and it's just about adding a category: "advanced" to the related struct field. For example:

mimir/pkg/util/validation/limits.go

Line 70 in dd1c12a

DropLabels flagext.StringSlice `yaml:"drop_labels" json:"drop_labels" category:"advanced"`

Alertmanager's firewall config is currently per-tenant:

mimir/pkg/util/validation/limits.go

Lines 131 to 132 in dd1c12a

	AlertmanagerReceiversBlockCIDRNetworks flagext.CIDRSliceCSV `yaml:"alertmanager_receivers_firewall_block_cidr_networks" json:"alertmanager_receivers_firewall_block_cidr_networks"`
	AlertmanagerReceiversBlockPrivateAddresses bool `yaml:"alertmanager_receivers_firewall_block_private_addresses" json:"alertmanager_receivers_firewall_block_private_addresses"`

It would be great to make the proxy support config on a per-tenant basis too.

[supergicko]

Thx for clarification @pracucci

My next q for guidance would be following,

assuming i set up limits like

AlertmanagerReceiversAllowNoProxy bool yaml:"alertmanager_receivers_allow_no_proxy" json:"alertmanager_receivers_allow_no_proxy" category:"advanced"

and define them in multitenant.go

AlertmanagerReceiversAllowNoProxy(tenant string) bool

Whats the best way to pass this down to the api in

mimir/pkg/alertmanager/api.go

Line 415 in dd1c12a

func validateReceiverHTTPConfig(cfg commoncfg.HTTPClientConfig) error {

Passing down the limits to the generic function?

func validateReceiverHTTPConfig(cfg commoncfg.HTTPClientConfig, limits Limits)

which also would touch

mimir/pkg/alertmanager/api.go

Line 319 in dd1c12a

func validateAlertmanagerConfig(cfg interface{}) error {

Updating the global HTTPClientConfig seems too much

Br

[replay]

If this should be configurable on a per-tenant basis then the best place to enforce it might not be in the alertmanager api.
Instead I would check out how the firewall rules which @pracucci has linked to are implemented, would it be possible to allow/disallow http proxies in the same way?

[supergicko]

after reading cortexproject/cortex#4129 (comment) and GHSA-m45g-f45x-vv22 again i am not sure if disabling setting the proxy_url when saving it via api is needed anymore. If you already disable via api the usage of "The HTTP basic auth password_file", then its ok regarding the CVE - or am i missing something? @replay @pracucci

[pracucci]

Thanks @supergicko for the follow up. You made me change my mind. What I didn't consider previously, is that the current "firewall" also protects requests to the HTTP proxy (I've added a test in #2076), so I think we can just always allow proxy_url.

[supergicko]

@pracucci great i will update the PR #2050 to allow it in all configs in alertmanager.

[pracucci]

What I didn't consider previously, is that the current "firewall" also protects requests to the HTTP proxy (I've added a test in #2076), so I think we can just always allow proxy_url.

After further investigation, the OAuth2 HTTP proxy_url is not covered by our firewall, so I've keep it not allowed in #2317 for now.