New context for notifications

[rajagopalanand]

There is a timeout for the entire pipeline which is set here https://github.com/prometheus/alertmanager/blob/main/dispatch/dispatch.go#L445. The timeout func is defined here https://github.com/prometheus/alertmanager/blob/main/cmd/alertmanager/main.go#L397-L402.

However, there is no timeout set for the http calls made within each receiver. For example, in PagerDuty.go, it uses the same context from the pipeline. If a http call takes a while to respond, then notify would hang until the pipeline context times out which is set to group_interval (5m default). This also means there is no retry occurring in these cases

Is there a reason why there is not a separate context with timeout just for the http call?

[grobinson-grafana]

This is a bug in my opinion, and was recently fixed for webhooks in #4137.

[eyazici90]

Ahh, I have drafted an PRfeat: add timeout option for pagerduty notifier and then realized this issue 🤦 Glad to see that it is not only me :D

Feel free to review it please 🙏

I would like to add the same option for slack notifications afterwards as well .

Here it goes feat: add timeout option for slack notifier.