Repository navigation
[FEATURE] Blocking alert: threshold on total blocked wait time (level-triggered), not just event count #1839
Description
Which component(s) does this affect?
- Lite
- Darling
- Documentation
Problem Statement
Hi Erik,
Billions of thanks for your work!
Written with ai but checked by brilliant brains:
We've been going through AlertEngine.cs / RollingCountAlertGate.cs to understand how the Blocking and Deadlocks alerts behave, and we'd like to propose a change specifically for Blocking Detected — we think the current mechanism is well suited for Deadlocks, but not for blocking.
Current behavior (both Blocking and Deadlocks)
Both checks share the same edge-trigger gate (RollingCountAlertGate) over a 1-hour rolling window: an alert fires only when the current event count climbs above a watermark (i.e., a genuinely new event), and the watermark resets to 0 once the count drops back below the threshold. This makes a lot of sense for deadlocks — a deadlock is a discrete, always-significant event, and without the watermark a single deadlock lingering in the 1-hour window would re-fire an alert every cooldown cycle for the whole hour.
Why it doesn't fit Blocking
Blocking is a different kind of signal. In production environments, short blocks (10-15 seconds) happen routinely and are rarely worth an alert if they resolve on their own — but the count-based threshold doesn't distinguish a handful of trivial, short-lived blocks from one (or several) genuinely long, damaging block. Right now the check thresholds on how many blocked-process reports/rows are in the window, not on how long anything was actually blocked — so we either get noise from routine short blocks, or we set the count threshold high enough to be quiet and then miss the case that actually matters: one session blocked for several minutes.
When monitoring a large number of instances, this distinction matters a lot — we want to be paged when something is actually bad, not every time transient contention shows up.
Thanks!
Romanian dbas.
Proposed Solution
Proposed change
For the Blocking check, replace (or add as an alternative mode) a threshold on the sum of wait time across the currently blocked sessions, in seconds, instead of (or in addition to) the event count. For example: alert when the total blocking wait time across all currently blocked sessions exceeds 600 seconds — regardless of whether that's one session blocked for 600s or several summing to 600s.
Behaviorally, we'd like this to work like the Long-Running Query / Poison Wait / CPU checks already do (a simple level-check + cooldown, not the rolling-window edge-trigger): while the cumulative wait time stays above the threshold, keep sending the alert every cooldown cycle with the current blocking status/context (same content as today's Blocking Detected alert), so we can see how the situation is evolving — and stop only once it drops back below the threshold, at which point the alert resolves/clears as usual.
To be clear: we think the watermark/rolling-count mechanism should stay exactly as-is for Deadlocks — this proposal is scoped to Blocking only.
Happy to discuss threshold defaults, whether this should be a new setting (BlockingWaitSecondsThreshold) alongside the existing count threshold, or a full replacement of the count-based check.
Use Case
We monitor a large number of SQL Server instances (dozens+) with this application, so alert noise has a real cost — every unnecessary email trains us to ignore the tool.
Concrete example: on one of our production servers, short blocks of (example) 10-15 seconds happen several times an hour as part of normal contention (lock waits that clear on their own). With the current count-based threshold, we either get alerted on these routine blocks (noise) or we raise the threshold high enough to ignore them and risk missing a real problem.
What we actually want: get an email when the total blocking wait time across all currently blocked sessions ("children") on a server adds up to more than, say, 600 seconds — whether that's one session stuck for 10 minutes, or five sessions each blocked for 2 minutes at the same time. That total is a much better signal of "this needs someone's attention" than a raw count of blocked-process rows.
Once that 600-second threshold is crossed, we'd want the alert to keep firing every cooldown cycle (e.g. every 5 minutes) with the current blocking status — same as it does today — so we can watch the incident evolve (is it getting worse, is a new session joining the chain, etc.) instead of getting a single notification and then having to log in to check manually. When the total wait time drops back below the threshold, the alert should resolve/clear as usual.
Alternatives Considered
No response
Additional Context
No response
Activity
erikdarlingdata commentedon Jul 30, 2026
This is one of the best-reasoned feature requests this repo has received, and your reading of the code is correct on every point: blocking and deadlocks share the RollingCountAlertGate edge trigger over the 1-hour window, the watermark exists so a lingering event cannot re-fire for a whole hour, and that mechanism fits deadlocks (discrete, always significant) much better than blocking (continuous, severity lives in duration, not occurrence count).
I agree with the diagnosis and with the scoping. A count of blocked-process rows measures how often blocking happened; the sum of wait time measures how much it hurts. For a fleet of instances, the second is the pageable signal. And deadlocks keep the edge trigger exactly as-is.
Two things make this cheap to build, which is the good news:
Direction I'd take it: an additive setting (BlockingWaitSecondsThreshold, off by default so nobody's existing alerting changes out from under them) alongside the count threshold, level-triggered with the standard cooldown and resolution. When both thresholds are configured, either crossing fires - you would set the count threshold high or off and lean on wait time, which matches your use case.
One design question back at you, since you clearly know your workload: for "total blocking wait time across currently blocked sessions," would you rather the sum be computed over the sessions blocked right now (the latest snapshot only - a pure severity level, drops to zero the moment the chain clears), or over the blocking observed in the last few minutes (smoother, catches a chain that churns SPIDs but never quite clears)? The current-snapshot version is what your example describes and is my default answer; the windowed version resists a flapping chain. Both are buildable from the same rows.
No promised date, but this fits the engine cleanly and it is on the list. Thanks for doing the reading before filing - it shows, and it made this a design conversation instead of an archaeology project.
[-][FEATURE][/-][+][FEATURE] Blocking alert: threshold on total blocked wait time (level-triggered), not just event count[/+]sdinu13 commentedon Jul 30, 2026
I would rather have the sum be computed over the sessions blocked right now (the latest snapshot only - a pure severity level, drops to zero the moment the chain clears). Which means the total sum of the locks at the snapshot moment. The second version might fool us.
erikdarlingdata commentedon Jul 30, 2026
Snapshot-sum it is - that settles the last open design point, and your instinct about the windowed version is right: smoothing a severity signal is how you end up staring at a graph instead of getting paged.
Locked in: a new BlockingWaitSecondsThreshold setting, off by default, alongside the existing count threshold. Level-triggered on the sum of wait time across the sessions blocked at the latest snapshot; re-fires every cooldown with current context while above; resolves the moment the sum drops below. Deadlocks keep the edge trigger untouched. Work is underway - it will land on the dev branch first, and I will note it here when it does.
Add a blocking alert on total blocked wait time (#1839)
Merge pull request #1840 from erikdarlingdata/feature/1839-blocking-w…
erikdarlingdata commentedon Jul 30, 2026
Landed on dev (#1840), ahead of schedule. What shipped, exactly as we settled it:
One guard worth knowing about: the check refuses to fire or stay active on a stale snapshot, so a collection outage cannot page you with frozen data.
It will be in the next release; if you want it before then, tonight's nightly build off dev carries it: https://github.com/erikdarlingdata/PerformanceMonitor/releases/tag/nightly - set the threshold, and I would genuinely value hearing how the 600s default behaves across your fleet before this ships in a numbered release.
erikdarlingdata commentedon Jul 31, 2026
Shipped in #1840, on dev and in the nightly since 2026-07-30. Closing as landed - please open a fresh issue (or comment here, it can be reopened) with your fleet's experience of the 600s threshold once you have run it in anger; that feedback still shapes the default before the numbered release.
sdinu13 commentedon Jul 31, 2026
Hello, Erik!
I installed the latest nightly (30.07)
We havent't yet found where to set the 600 "Blocking Wait Threshold" in settings or anywhere we thought of searching through the app files.
We actually found another thing worth changing, it's a 200 cap for which we don't find an actual sense (in the case of locks), we found it in the settings yesterday hardcoded as "DefaultCap" or something like that. The issue with is that, once it reaches this cap, the mails/alerts stop being fired/sent, which basically is opposite to the purpose of a long running locks. When you have 100 blocked sessions, and this cap is enabled, after two 5 seconds scan of the app, the cap is reached.
We would recomend to be able to set both these settings, somewhere.
Thanks!
erikdarlingdata#1912: Lite half — collapse split slices in the hot st…
Merge upstream/dev: integrate upstream changes with PagerDuty webhook
Lite's MCP reports every alert group Darling's does