Repository navigation
email_cooldown_minutes is the only throttle on Slack/Teams/PagerDuty delivery, but it is named for email, absent from get_alert_settings/update_alert_settings, and capped at 120 minutes #3314
Description
Summary
The only throttle between a fired alert and a Slack / Teams / PagerDuty / generic-webhook post is email_cooldown_minutes:
WebhookAlertService.cs:167 TimeSpan.FromMinutes(_settings.EmailCooldownMinutes)
EmailSendCore.cs:121 TimeSpan.FromMinutes(_settings.EmailCooldownMinutes)
Three problems, all operability rather than correctness:
1. It is named for the one channel an operator may not be using. A headless deployment with Slack configured and no SMTP at all is still governed entirely by a setting called email_cooldown_minutes, living in the SMTP config block (DarlingConfig.Smtp). There is no reason to look there when tuning Slack volume.
2. It is not reachable from the control plane. get_alert_settings does not return it and update_alert_settings does not accept it — the accepted-key switch in DarlingMcpAlertTools carries cooldown_minutes (the engine's fire cooldown) and analysis_notify_cooldown_minutes, but not this one. The column exists in config.config_notification and is written only by ViewerDataService.Notification, i.e. the WPF Settings window. So on a headless box the sole documented path to the setting that governs channel volume is a desktop app, or a hand-written UPDATE.
This also makes it invisible: an operator reading get_alert_settings sees cooldown_minutes: 5 and reasonably concludes that is the delivery cadence. It is not — it gates AlertEngine's fire decision. The delivery cadence is a different number in a different table that the tool will not show them.
3. The 120-minute clamp is low for a fleet. DarlingAlertSettings.cs:230 clamps to 1..120. On a fleet where one recurring signature is legitimately open for hours, 2 hours is the longest an operator can ask for, per fingerprint.
Observed
Measured on a 42-server store with Slack configured and no SMTP: 18 Deadlocks Detected cards in 74 minutes. The setting was at its default 15, never explicitly configured, and the delivered/suppressed boundary in config_alert_log lands on it exactly:
| fingerprint gap since its own last delivery | notification_type |
|---|---|
| 11.5 min | undelivered |
| 29.5 min | webhook |
| 49.2 min | webhook |
That is the 15-minute per-fingerprint window behaving exactly as #1154 specified. The defect is not the mechanism; it is that the number driving it cannot be found or changed through the interface the deployment actually uses.
Suggested
- Accept and return it in
update_alert_settings/get_alert_settingsunder a channel-neutral name (delivery.cooldown_minutes), keepingemail_cooldown_minutesas the stored column and an accepted alias so existing configs and the Lite Settings window keep working. - Raise the clamp ceiling, or document why 120 is the ceiling.
get_alert_settingsshould return it next todelivery.mode/delivery.per_event_max, where someone tuning volume will actually look.
Notes
cooldown_minutes and email_cooldown_minutes being two different numbers with near-identical names, governing two different stages of the same pipeline, is the root of the confusion — worth a sentence in each one's description regardless of whether the rename happens.
Activity
erikdarlingdata commentedon Sep 12, 2026
Claude posting for Erik Darling
Fixed by #3336, merged to
dev. Closing explicitly — closing keywords no-op ondev.The delivery cooldown is now reachable from the control plane as
delivery.cooldown_minutes, besidedelivery.modeanddelivery.per_event_maxwhere someone tuning channel volume will actually look.get_alert_settingsreports it andupdate_alert_settingsaccepts it;email_cooldown_minutesstays the stored column and a write-only alias, and sending both spellings is refused naming both.The 120-minute ceiling stays, deliberately, and the argument is better than the one in my filing. The MCP bound has to equal the engine clamp or the tool accepts a value the engine silently rewrites — a trap already pinned twice — so raising it is a four-site cross-SKU behaviour change. More importantly the knob is the wrong shape for the need: it is one global number applied to every fingerprint on every server, where the thing an operator actually wants is to quieten one signature.
create_mute_ruledoes that scoped, expiring and disclosed, and the tool description now points there.One grant moved, column-level and measured.
GRANT UPDATE (email_cooldown_minutes) ON config.config_notification TO mcp. That table holds the SMTP password blob, the webhook URLs and the PagerDuty routing key, so the grant is scoped to the single non-secret column, and a live test asserts 42501 onsmtp_encrypted_password,slack_urland the non-secret siblingsmtp_host— proving the grant is one column wide rather than merely excluding secrets. The isolation was necessary because a missing column SELECT and a missing column UPDATE raise the identical 42501.No migration rung: the column already existed.
One review finding, fixed rather than filed: the two reads across the two tables were unsynchronized, so a payload could mix pre- and post-update state while the write path took a transaction specifically to prevent that. Now one
REPEATABLE READsnapshot, with the single-table reads made private so reintroducing the split is a compile error.Merge pull request #3340 from erikdarlingdata/chore/changelog-3314
Add the erikdarlingdata#3314 CHANGELOG entry for the control-plane de…