Log the GroupKey and alerts in retry

[grobinson-grafana]

What this PR does

This pull request updates notify.go to log the GroupKey and fingerprints of an alert at the debug level, and just the GroupKey at the warning level should the notify attempt fail.

Here is an example of it at debug level:

ts=2023-08-03T09:28:36.829Z caller=dispatch.go:163 level=debug component=dispatcher msg="Received alert" alert=foo[c5a7cb1][active]
ts=2023-08-03T09:28:38.725Z caller=dispatch.go:163 level=debug component=dispatcher msg="Received alert" alert=bar[4656093][active]
ts=2023-08-03T09:28:51.830Z caller=dispatch.go:515 level=debug component=dispatcher aggrGroup="{}:{alertname=\"foo\"}" msg=flushing alerts=[foo[c5a7cb1][active]]
ts=2023-08-03T09:28:51.984Z caller=notify.go:757 level=debug component=dispatcher receiver=email integration=email[0] aggrGroup="{}:{alertname=\"foo\"}" alerts=[foo[c5a7cb1][active]] msg="Notify success" attempts=1
ts=2023-08-03T09:28:53.726Z caller=dispatch.go:515 level=debug component=dispatcher aggrGroup="{}:{alertname=\"bar\"}" msg=flushing alerts=[bar[4656093][active]]
ts=2023-08-03T09:28:53.855Z caller=notify.go:757 level=debug component=dispatcher receiver=email integration=email[0] aggrGroup="{}:{alertname=\"bar\"}" alerts=[bar[4656093][active]] msg="Notify success" attempts=1

and when notify attempt fails:

ts=2023-08-03T09:34:42.505Z caller=notify.go:745 level=warn component=dispatcher receiver=email integration=email[0] aggrGroup="{}:{alertname=\"foo\"}" msg="Notify attempt failed, will retry later" attempts=1 err="establish connection to server: dial tcp 127.0.0.1:1025: connect: connection refused"

Motivation

Without this change it's almost impossible to correlate a Notify success log line with the aggregation group's flush log line, and vice versa. This change will make it easier to debug issues around missing notifications or unexpected notifications for alerts when running large Alertmanager installations with 10,000s of alerts.

For example:

ts=2023-08-03T09:45:57.934Z caller=dispatch.go:515 level=debug component=dispatcher aggrGroup="{}:{alertname=\"foo\"}" msg=flushing alerts=[foo[c5a7cb1][active]]
ts=2023-08-03T09:45:58.139Z caller=notify.go:752 level=debug component=dispatcher receiver=email integration=email[0] msg="Notify success" attempts=1
ts=2023-08-03T09:45:59.801Z caller=dispatch.go:515 level=debug component=dispatcher aggrGroup="{}:{alertname=\"bar\"}" msg=flushing alerts=[bar[4656093][active]]
ts=2023-08-03T09:45:59.935Z caller=notify.go:752 level=debug component=dispatcher receiver=email integration=email[0] msg="Notify success" attempts=1

[simonpasquier]

lgtm

[gotjosh]

Rebased to make sure CI passes.

[gotjosh]

LGTM

[gotjosh]

Thank you very much for your contribution.