#3513: Mark muted alerts #3793

grobinson-grafana · 2024-04-02T13:54:44Z

This pull request updates TimeMuteStage and TimeActiveStage to mark groups as muted when its alerts are muted by an active or mute time interval, and remove any existing markers when outside all active and mute time intervals.

It is based on #3792, #3794 and #3795.

grobinson-grafana · 2024-04-03T12:29:00Z

dispatch/dispatch.go

@@ -182,6 +184,7 @@ func (d *Dispatcher) run(it provider.AlertIterator) {
 				for _, ag := range groups {
 					if ag.empty() {
 						ag.stop()
+						d.marker.DeleteByGroupKey(ag.GroupKey())


We need to delete the marker when the aggregation group is deleted. There is a similar mechanism in provider/mem/mem.go:

alertmanager/provider/mem/mem.go

Line 108 in 14cbe63

m.Delete(alert.Fingerprint())

This commit updates TimeMuteStage and TimeActiveStage to mark groups as muted when its alerts are muted by an active or mute time interval, and remove any existing markers when outside all active and mute time intervals. Signed-off-by: George Robinson <george.robinson@grafana.com>

grobinson-grafana · 2024-04-30T15:04:19Z

dispatch/dispatch.go

@@ -107,7 +108,7 @@ func NewDispatcher(
 	ap provider.Alerts,
 	r *Route,
 	s notify.Stage,
-	mk types.AlertMarker,


mk was never used, here I am replacing it with types.GroupMarker.

grobinson-grafana · 2024-04-30T15:04:58Z

dispatch/dispatch.go

@@ -145,8 +147,8 @@ func (d *Dispatcher) Run() {
 }

 func (d *Dispatcher) run(it provider.AlertIterator) {
-	cleanup := time.NewTicker(30 * time.Second)
-	defer cleanup.Stop()
+	maintenance := time.NewTicker(30 * time.Second)


I renamed this as all other "cleanup" functions in Alertmanager are called maintenance.

grobinson-grafana · 2024-04-30T15:05:16Z

dispatch/dispatch.go

@@ -175,28 +177,30 @@ func (d *Dispatcher) run(it provider.AlertIterator) {
 			}
 			d.metrics.processingDuration.Observe(time.Since(now).Seconds())

-		case <-cleanup.C:


I moved this logic to a method called doMaintenance. The main purpose of this was being able to test it.

grobinson-grafana · 2024-04-30T15:06:13Z

dispatch/dispatch.go

+		for _, ag := range groups {
+			if ag.empty() {
+				ag.stop()
+				d.marker.DeleteByGroupKey(ag.routeID, ag.GroupKey())


We delete the marker for the aggregation group when the aggregation group is deleted itself.

grobinson-grafana · 2024-04-30T15:06:39Z

dispatch/dispatch.go

@@ -374,6 +378,7 @@ type aggrGroup struct {
 	labels   model.LabelSet
 	opts     *RouteOpts
 	logger   log.Logger
+	routeID  string


We needed to add routeID to aggrGroup because it's used in d.marker.DeleteByGroupKey(ag.routeID, ag.GroupKey()).

grobinson-grafana · 2024-04-30T15:07:10Z

dispatch/dispatch.go

@@ -447,6 +453,7 @@ func (ag *aggrGroup) run(nf notifyFunc) {
 			ctx = notify.WithRepeatInterval(ctx, ag.opts.RepeatInterval)
 			ctx = notify.WithMuteTimeIntervals(ctx, ag.opts.MuteTimeIntervals)
 			ctx = notify.WithActiveTimeIntervals(ctx, ag.opts.ActiveTimeIntervals)
+			ctx = notify.WithRouteID(ctx, ag.routeID)


We also need to add it to the context so it can be extracted in TimeMuteStage and TimeActiveStage.

grobinson-grafana · 2024-04-30T15:07:34Z

dispatch/dispatch_test.go

@@ -691,3 +691,48 @@ type limits struct {
 func (l limits) MaxNumberOfAggregationGroups() int {
 	return l.groups
 }
+
+func TestDispatcher_DoMaintenance(t *testing.T) {


Here I am testing that the marker is deleted when the aggregation group is garbage collected.

grobinson-grafana · 2024-04-30T15:09:15Z

notify/notify_test.go

@@ -827,12 +828,6 @@ func TestTimeMuteStage(t *testing.T) {
 	}
 	eveningsAndWeekends := map[string][]timeinterval.TimeInterval{
 		"evenings": {{
-			Weekdays: []timeinterval.WeekdayRange{{


I removed Weekdays so I could test a group being muted by both evenings and weekends at the same time.

grobinson-grafana · 2024-04-30T15:09:48Z

notify/notify_test.go


 			// Get the names of all time intervals for the context.
 			muteTimeIntervalNames := make([]string, 0, len(test.intervals))
 			for name := range test.intervals {
 				muteTimeIntervalNames = append(muteTimeIntervalNames, name)
 			}
+			// Sort the names so we can compare mutedBy with test.mutedBy.
+			sort.Strings(muteTimeIntervalNames)


Needs to be sorted because test.intervals is a map.

grobinson-grafana · 2024-04-30T15:14:02Z

notify/notify_test.go

+				require.NoError(t, prom_testutil.GatherAndCompare(r, strings.NewReader(`
+# HELP alertmanager_marked_alerts How many alerts by state are currently marked in the Alertmanager regardless of their expiry.
+# TYPE alertmanager_marked_alerts gauge
+alertmanager_marked_alerts{state="active"} 0


Do I need to add metrics for marked groups? @gotjosh

I don't think so - I wouldn't really understand their historically use-case.

gotjosh

LGTM

dispatch/dispatch.go

gotjosh · 2024-05-09T14:46:42Z

notify/notify.go

+// RouteID extracts a RouteID from the context. Iff none exists, the
+// // second argument is false.


Suggested change

// RouteID extracts a RouteID from the context. Iff none exists, the

// // second argument is false.

// RouteID extracts a RouteID from the context. If none exists, the

// second argument is false.

The other comments have the same typo of iff

Oh iff is not a spelling mistake, it refers to if and only if.

Today I learned 🤯

Signed-off-by: George Robinson <george.robinson@grafana.com>

* Mark muted groups This commit updates TimeMuteStage and TimeActiveStage to mark groups as muted when its alerts are muted by an active or mute time interval, and remove any existing markers when outside all active and mute time intervals. Signed-off-by: George Robinson <george.robinson@grafana.com> * Remove unlock to defer Signed-off-by: George Robinson <george.robinson@grafana.com> --------- Signed-off-by: George Robinson <george.robinson@grafana.com>

* [CHANGE] Deprecate and remove api/v1/ #2970 * [CHANGE] Remove unused feature flags #3676 * [CHANGE] Newlines in smtp password file are now ignored #3681 * [CHANGE] Change compat metrics to counters #3686 * [CHANGE] Do not register compat metrics in amtool #3713 * [CHANGE] Remove metrics from compat package #3714 * [CHANGE] Mark muted alerts #3793 * [FEATURE] Add metric for inhibit rules #3681 * [FEATURE] Support UTF-8 label matchers #3453, #3507, #3523, #3483, #3567, #3568, #3569, #3571, #3595, #3604, #3619, #3658, #3659, #3662, #3668, 3572 * [FEATURE] Add counter to track alerts dropped outside of time_intervals #3565 * [FEATURE] Add date and tz functions to templates #3812 * [FEATURE] Add limits for silences #3852 * [FEATURE] Add time helpers for templates #3863 * [FEATURE] Add auto GOMAXPROCS #3837 * [FEATURE] Add auto GOMEMLIMIT #3895 * [FEATURE] Add Jira receiver integration #3590 * [ENHANCEMENT] Add the receiver name to notification metrics #3045 * [ENHANCEMENT] Add the route ID to uuid #3372 * [ENHANCEMENT] Add duration to the notify success message #3559 * [ENHANCEMENT] Implement webhook_url_file for discord and msteams #3555 * [ENHANCEMENT] Add debug logs for muted alerts #3558 * [ENHANCEMENT] API: Allow the Silences API to use their own 400 response #3610 * [ENHANCEMENT] Add summary to msteams notification #3616 * [ENHANCEMENT] Add context reasons to notifications failed counter #3631 * [ENHANCEMENT] Add optional native histogram support to latency metrics #3737 * [ENHANCEMENT] Enable setting ThreadId for Telegram notifications #3638 * [ENHANCEMENT] Allow webex roomID from template #3801 * [BUGFIX] Add missing integrations to notify metrics #3480 * [BUGFIX] Add missing ttl in pushhover #3474 * [BUGFIX] Fix scheme required for webhook url in amtool #3409 * [BUGFIX] Remove duplicate integration from metrics #3516 * [BUGFIX] Reflect Discord's max length message limits #3597 * [BUGFIX] Fix nil error in warn logs about incompatible matchers #3683 * [BUGFIX] Fix a small number of inconsistencies in compat package logging #3718 * [BUGFIX] Fix log line in featurecontrol #3719 * [BUGFIX] Fix panic in acceptance tests #3592 * [BUGFIX] Fix flaky test TestClusterJoinAndReconnect/TestTLSConnection #3722 * [BUGFIX] Fix crash on errors when url_file is used #3800 * [BUGFIX] Fix race condition in dispatch.go #3826 * [BUGFIX] Fix race conditions in the memory alerts store #3648 * [BUGFIX] Hide config.SecretURL when the URL is incorrect. #3887 * [BUGFIX] Fix invalid silence causes incomplete updates #3898 * [BUGFIX] Fix leaking of Silences matcherCache entries #3930 * [BUGFIX] Close SMTP submission correctly to handle errors #4006 Signed-off-by: SuperQ <superq@gmail.com>

* Release v0.28.0-rc.0 * [CHANGE] Templating errors in the SNS integration now return an error. #3531 #3879 * [FEATURE] Add a new Microsoft Teams integration based on Flows #4024 * [FEATURE] Add a new Rocket.Chat integration #3600 * [FEATURE] Add a new Jira integration #3590 #3931 * [FEATURE] Add support for `GOMEMLIMIT`, enable it via the feature flag `--enable-feature=auto-gomemlimit`. #3895 * [FEATURE] Add support for `GOMAXPROCS`, enable it via the feature flag `--enable-feature=auto-gomaxprocs`. #3837 * [FEATURE] Add support for limits of silences including the maximum number of active and pending silences, and the maximum size per silence (in bytes). You can use the flags `--silences.max-silences` and `--silences.max-silence-size-bytes` to set them accordingly #3852 #3862 #3866 #3885 #3886 #3877 * [FEATURE] Muted alerts now show whether they are suppressed or not in both the `/api/v2/alerts` endpoint and the Alertmanager UI. #3793 #3797 #3792 * [ENHANCEMENT] Add support for `content`, `username` and `avatar_url` in the Discord integration. `content` and `username` also support templating. #4007 * [ENHANCEMENT] Only invalidate the silences cache if a new silence is created or an existing silence replaced - should improve latency on both `GET api/v2/alerts` and `POST api/v2/alerts` API endpoint. #3961 * [ENHANCEMENT] Add image source label to Dockerfile. To get changelogs shown when using Renovate #4062 * [ENHANCEMENT] Build using go 1.23 #4071 * [ENHANCEMENT] Support setting a global SMTP TLS configuration. #3732 * [ENHANCEMENT] The setting `room_id` in the WebEx integration can now be templated to allow for dynamic room IDs. #3801 * [ENHANCEMENT] Enable setting `message_thread_id` for the Telegram integration. #3638 * [ENHANCEMENT] Support the `since` and `humanizeDuration` functions to templates. This means users can now format time to more human-readable text. #3863 * [ENHANCEMENT] Support the `date` and `tz` functions to templates. This means users can now format time in a specified format and also change the timezone to their specific locale. #3812 * [ENHANCEMENT] Latency metrics now support native histograms. #3737 * [BUGFIX] Fix the SMTP integration not correctly closing an SMTP submission, which may lead to unsuccessful dispatches being marked as successful. #4006 * [BUGFIX] The `ParseMode` option is now set explicitly in the Telegram integration. If we don't HTML tags had not been parsed by default. #4027 * [BUGFIX] Fix a memory leak that was caused by updates silences continuously. #3930 * [BUGFIX] Fix hiding secret URLs when the URL is incorrect. #3887 * [BUGFIX] Fix a race condition in the alerts - it was more of a hypothetical race condition that could have occurred in the alert reception pipeline. #3648 * [BUGFIX] Fix a race condition in the alert delivery pipeline that would cause a firing alert that was delivered earlier to be deleted from the aggregation group when instead it should have been delivered again. #3826 * [BUGFIX] Fix version in APIv1 deprecation notice. #3815 * [BUGFIX] Fix crash errors when using `url_file` in the Webhook integration. #3800 * [BUGFIX] fix `Route.ID()` returns conflicting IDs. #3803 * [BUGFIX] Fix deadlock on the alerts memory store. #3715 * [BUGFIX] Fix `amtool template render` when using the default values. #3725 * [BUGFIX] Fix `webhook_url_file` for both the Discord and Microsoft Teams integrations. #3728 #3745 --------- Signed-off-by: SuperQ <superq@gmail.com> Signed-off-by: gotjosh <josue.abreu@gmail.com> Co-authored-by: gotjosh <josue.abreu@gmail.com>

* [CHANGE] Templating errors in the SNS integration now return an error. #3531 #3879 * [CHANGE] Adopt log/slog, drop go-kit/log #4089 * [FEATURE] Add a new Microsoft Teams integration based on Flows #4024 * [FEATURE] Add a new Rocket.Chat integration #3600 * [FEATURE] Add a new Jira integration #3590 #3931 * [FEATURE] Add support for `GOMEMLIMIT`, enable it via the feature flag `--enable-feature=auto-gomemlimit`. #3895 * [FEATURE] Add support for `GOMAXPROCS`, enable it via the feature flag `--enable-feature=auto-gomaxprocs`. #3837 * [FEATURE] Add support for limits of silences including the maximum number of active and pending silences, and the maximum size per silence (in bytes). You can use the flags `--silences.max-silences` and `--silences.max-silence-size-bytes` to set them accordingly #3852 #3862 #3866 #3885 #3886 #3877 * [FEATURE] Muted alerts now show whether they are suppressed or not in both the `/api/v2/alerts` endpoint and the Alertmanager UI. #3793 #3797 #3792 * [ENHANCEMENT] Add support for `content`, `username` and `avatar_url` in the Discord integration. `content` and `username` also support templating. #4007 * [ENHANCEMENT] Only invalidate the silences cache if a new silence is created or an existing silence replaced - should improve latency on both `GET api/v2/alerts` and `POST api/v2/alerts` API endpoint. #3961 * [ENHANCEMENT] Add image source label to Dockerfile. To get changelogs shown when using Renovate #4062 * [ENHANCEMENT] Build using go 1.23 #4071 * [ENHANCEMENT] Support setting a global SMTP TLS configuration. #3732 * [ENHANCEMENT] The setting `room_id` in the WebEx integration can now be templated to allow for dynamic room IDs. #3801 * [ENHANCEMENT] Enable setting `message_thread_id` for the Telegram integration. #3638 * [ENHANCEMENT] Support the `since` and `humanizeDuration` functions to templates. This means users can now format time to more human-readable text. #3863 * [ENHANCEMENT] Support the `date` and `tz` functions to templates. This means users can now format time in a specified format and also change the timezone to their specific locale. #3812 * [ENHANCEMENT] Latency metrics now support native histograms. #3737 * [ENHANCEMENT] Add timeout option for webhook notifier. #4137 * [BUGFIX] Fix the SMTP integration not correctly closing an SMTP submission, which may lead to unsuccessful dispatches being marked as successful. #4006 * [BUGFIX] The `ParseMode` option is now set explicitly in the Telegram integration. If we don't HTML tags had not been parsed by default. #4027 * [BUGFIX] Fix a memory leak that was caused by updates silences continuously. #3930 * [BUGFIX] Fix hiding secret URLs when the URL is incorrect. #3887 * [BUGFIX] Fix a race condition in the alerts - it was more of a hypothetical race condition that could have occurred in the alert reception pipeline. #3648 * [BUGFIX] Fix a race condition in the alert delivery pipeline that would cause a firing alert that was delivered earlier to be deleted from the aggregation group when instead it should have been delivered again. #3826 * [BUGFIX] Fix version in APIv1 deprecation notice. #3815 * [BUGFIX] Fix crash errors when using `url_file` in the Webhook integration. #3800 * [BUGFIX] fix `Route.ID()` returns conflicting IDs. #3803 * [BUGFIX] Fix deadlock on the alerts memory store. #3715 * [BUGFIX] Fix `amtool template render` when using the default values. #3725 * [BUGFIX] Fix `webhook_url_file` for both the Discord and Microsoft Teams integrations. #3728 #3745 * [BUGFIX] Fix wechat api link #4084 * [BUGFIX] Fix build info metric #4166 Signed-off-by: SuperQ <superq@gmail.com>

grobinson-grafana mentioned this pull request Apr 2, 2024

Muted alerts are not suppressed in API #3513

Closed

grobinson-grafana marked this pull request as draft April 2, 2024 18:20

grobinson-grafana force-pushed the grobinson/mark-muted-alerts-3 branch from 6baeccc to ff97f73 Compare April 3, 2024 10:18

grobinson-grafana marked this pull request as ready for review April 3, 2024 10:26

grobinson-grafana commented Apr 3, 2024

View reviewed changes

grobinson-grafana mentioned this pull request Apr 3, 2024

#3513: Show muted alerts in the Alert Groups API #3797

Merged

grobinson-grafana changed the title ~~Mark muted alerts~~ #3513: Mark muted alerts Apr 3, 2024

grobinson-grafana force-pushed the grobinson/mark-muted-alerts-3 branch 2 times, most recently from 2021070 to 9a433cf Compare April 11, 2024 14:32

grobinson-grafana marked this pull request as draft April 11, 2024 15:01

grobinson-grafana force-pushed the grobinson/mark-muted-alerts-3 branch 4 times, most recently from ff2401d to 3cec1e8 Compare April 29, 2024 16:55

grobinson-grafana marked this pull request as ready for review April 30, 2024 13:40

grobinson-grafana mentioned this pull request Apr 30, 2024

#3513: Add GroupMarker interface #3792

Merged

grobinson-grafana force-pushed the grobinson/mark-muted-alerts-3 branch from 07bdde5 to 8ffc8ae Compare April 30, 2024 14:51

grobinson-grafana force-pushed the grobinson/mark-muted-alerts-3 branch from 8ffc8ae to 9342bc2 Compare April 30, 2024 15:03

grobinson-grafana commented Apr 30, 2024

View reviewed changes

gotjosh approved these changes May 9, 2024

View reviewed changes

Remove unlock to defer

c921eab

Signed-off-by: George Robinson <george.robinson@grafana.com>

gotjosh merged commit c4a763c into prometheus:main May 13, 2024
11 checks passed

SuperQ mentioned this pull request Oct 16, 2024

Release v0.28.0-rc.0 #4072

Merged

SuperQ mentioned this pull request Dec 19, 2024

Release v0.28.0 #4176

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

#3513: Mark muted alerts #3793

#3513: Mark muted alerts #3793

grobinson-grafana commented Apr 2, 2024 •

edited

Loading

grobinson-grafana Apr 3, 2024

grobinson-grafana Apr 30, 2024

grobinson-grafana Apr 30, 2024

grobinson-grafana Apr 30, 2024

grobinson-grafana Apr 30, 2024

grobinson-grafana Apr 30, 2024

grobinson-grafana Apr 30, 2024

grobinson-grafana Apr 30, 2024

grobinson-grafana Apr 30, 2024

grobinson-grafana Apr 30, 2024

grobinson-grafana Apr 30, 2024

gotjosh May 9, 2024

gotjosh left a comment

gotjosh May 9, 2024

grobinson-grafana May 9, 2024

gotjosh May 10, 2024

		// RouteID extracts a RouteID from the context. Iff none exists, the
		// // second argument is false.

#3513: Mark muted alerts #3793

#3513: Mark muted alerts #3793

Conversation

grobinson-grafana commented Apr 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gotjosh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

grobinson-grafana commented Apr 2, 2024 •

edited

Loading