Retry HTTP requests on "context deadline exceede" #1227

It would be nice if notification-controller would retry requests in case we see a "context deadline exceeded" error. We mainly use Alertmanager for dispatching notifications and in case Alertmanager is shortly unavailable (just a few seconds on node shift or the like), we immediately see these errors:

postMessage failed: failed to execute request: POST http://prometheus-stack-kube-prom-alertmanager.monitoring:9093/api/v2/alerts/ giving up after 1 attempt(s): context deadline exceeded

Using the retry mechanism here, too, would immediately get rid of the problem.