Repository navigation
SQL Server's Poison Wait alert measures accumulated starvation over a ten-minute window like its PostgreSQL twin, so one slow wait no longer pages and a THREADPOOL storm no longer sleeps (#3539 A4) - #3593
Conversation
… ten-minute window like its PostgreSQL twin (#3539 A4) The SQL Server Poison Wait alert judged ONE collector delta row's avg-ms-per-wait against PoisonWaitThresholdMs (500), presence-flat CRITICAL, no volume floor: one 600 ms wait paged and a THREADPOOL storm of thousands of 8 ms waits slept. The PostgreSQL twin (#2711) already fired on accumulated wait over a ten-minute window, graded Warning/Critical, under the same alert name, switch and mute key. Port the PostgreSQL shape to SQL Server. The read (both SKUs) SUMs delta_wait_time_ms / delta_waiting_tasks per poison wait type over the window with no task filter, no LIMIT and no threshold; the new shared PoisonWaitEvaluator grades the sum (Warning at 1.0 avg tasks stuck = 600 s / 10 min, Critical at 10) and both engines' constants are one definition (PostgresAlertEvaluator's are aliases). Fleet margin on the SQL side: the worst ten-minute bucket across 43 servers / 4 days was 0.0097 avg waiters, ~100x under the bar. The clear arm now requires an observed window (an empty read holds the alert; the retired shape announced Cleared on collector silence). PoisonWaitThresholdMs stays readable and persisted but is no longer consulted; docs and settings comments say so.
|
Nice, thorough port — the shared One gap: both live Settings UIs still describe the retired knob as if it's the live threshold, and nothing in this PR updates them:
After this change, This isn't a Lite-vs-Darling drift (both apps have the identical gap), but it is a real user-facing correctness issue introduced by redefining what the setting does without updating where it's surfaced. |
There was a problem hiding this comment.
Solid port of the accumulation-based Poison Wait shape to SQL Server — the shared evaluator, alias-not-duplicate constants (with a source-text pin against drift), and the SQL parity pin between Lite's DuckDB text and Darling's Postgres text are all well done, and the edge-case test coverage (freshness gating on firing types only, hold-vs-clear on empty vs. observed-quiet reads, worst-first mute keying) is thorough and correct as written.
- User-facing correctness bug (both SKUs): Neither Lite's nor Darling Viewer's Settings window was updated for the semantic change. Both still show a checkbox labeled
"Poison waits (THREADPOOL, etc.) avg wait ≥"next to an editable ms threshold, and both build an alert-preview line reading"poison waits >= {threshold}ms avg"(Lite/Windows/SettingsWindow.xaml:249/SettingsWindow.xaml.cs:950;Darling/PerformanceMonitor.Darling.Service/../Viewer/SettingsWindow.xaml:357/SettingsWindow.xaml.cs:1106). SincePoisonWaitThresholdMsis now explicitly unconsulted by the engine (per this PR's ownIAlertEngineSettingsdoc and README update), these controls actively mislead an operator into believing they can tune poison-wait sensitivity via that box, with zero effect and no UI indication that it's dead. Not a Lite/Darling parity drift (both apps have the identical gap) — a correctness gap left by the semantic change itself.
Everything else checked out: SQL is parameterized (no injection risk), the shared constants/tests correctly enforce cross-engine grading parity, and the freshness/mute/clear state machine changes are consistent with the codebase's existing patterns and covered by new tests.
…2E compares at the store's precision Review finding on #3593: both SKUs' Settings windows still labelled the poison checkbox "avg wait >=" beside an editable ms box, and the save-preview line read "poison waits >= Nms avg" - a knob the engine no longer consults. Label now says the alert measures accumulated wait over a 10-minute window and that the ms bar is retired; the box is disabled with a tooltip and kept so saved settings round-trip; the preview states the live bar from the shared constants. Identical edit on Lite and the Darling Viewer. CI: the live read-adapter E2E compared a seeded DateTime (100 ns ticks) to the value PostgreSQL stored at microsecond precision; compare at the store's precision and assert the in-window row's clock won separately.
|
Review finding 1 (Settings windows still offering the retired ms bar) — fixed in 67e97aa, both twins identically:
I had listed these surfaces as outside the lane's declared file boundary in the PR body; the review is right that a label my change falsifies is part of my change, so the boundary extension is taken and the "does NOT do" paragraph is stale on that point (the residuals that remain out of lane: Same push also fixes the one CI failure, mine: |
Finding 1 (Settings windows offering the retired ms bar) remediated in 67e97aa on both SKUs; see the PR comment for the disposition.
|
Reviewed the full diff (both commits). This is a well-scoped, thoroughly-tested port of the PostgreSQL Poison Wait accumulation shape to SQL Server. Notes:
Nothing rises to a blocking finding. |
There was a problem hiding this comment.
LGTM — reviewed the SQL Server Poison Wait accumulation-shape port (AlertEngine, PoisonWaitEvaluator, both read adapters, both Settings windows) for correctness, Lite/Darling parity, and SQL injection; no substantive issues found. Details in the review comment.
…ical to the retired read, ~30 rows per server in the head chunk), not by index
| <CheckBox x:Name="AlertPoisonWaitCheckBox" Content="Poison waits (THREADPOOL, etc.) — accumulated wait over a 10-minute window; legacy avg-ms bar (no longer used):" VerticalAlignment="Center" | ||
| Foreground="{DynamicResource ForegroundBrush}"/> | ||
| <TextBox x:Name="AlertPoisonWaitThresholdBox" Width="45" Text="500" Margin="6,0,0,0" VerticalAlignment="Center"/> | ||
| <TextBox x:Name="AlertPoisonWaitThresholdBox" Width="45" Text="500" Margin="6,0,0,0" VerticalAlignment="Center" IsEnabled="False" |
There was a problem hiding this comment.
This IsEnabled="False" is undone at runtime: UpdateAlertControlStates() in SettingsWindow.xaml.cs sets AlertPoisonWaitThresholdBox.IsEnabled = enabled (where enabled tracks the master AlertsEnabledCheckBox), and it's called both on window load and on every toggle of that checkbox. With alerts enabled — the default — the box re-enables itself as soon as the window opens, silently defeating the point of this change (letting an operator tune a setting the engine no longer reads). Same issue in the Darling Viewer's SettingsWindow.xaml.cs.
AlertPoisonWaitThresholdBox should be excluded from that loop (or forced disabled unconditionally there too).
There was a problem hiding this comment.
Fixed: UpdateAlertControlStates() now sets this box IsEnabled = false unconditionally (both SKUs), so the XAML disablement holds on load and on every master-switch toggle.
| <CheckBox x:Name="AlertPoisonWaitCheckBox" Content="Poison waits (THREADPOOL, etc.) — accumulated wait over a 10-minute window; legacy avg-ms bar (no longer used):" VerticalAlignment="Center" | ||
| Foreground="{DynamicResource ForegroundBrush}"/> | ||
| <TextBox x:Name="AlertPoisonWaitThresholdBox" Width="45" Text="500" Margin="6,0,0,0" VerticalAlignment="Center"/> | ||
| <TextBox x:Name="AlertPoisonWaitThresholdBox" Width="45" Text="500" Margin="6,0,0,0" VerticalAlignment="Center" IsEnabled="False" |
There was a problem hiding this comment.
Same issue as the Lite twin: IsEnabled="False" here is overridden at runtime by UpdateAlertControlStates() in the code-behind, which sets AlertPoisonWaitThresholdBox.IsEnabled = enabled on window load and on every AlertsEnabledCheckBox toggle. With alerts enabled (the default), the box re-enables as soon as the window opens, so the "retired, don't edit" disablement doesn't actually hold in the common case.
There was a problem hiding this comment.
Fixed: UpdateAlertControlStates() now sets this box IsEnabled = false unconditionally (both SKUs), so the XAML disablement holds on load and on every master-switch toggle.
There was a problem hiding this comment.
Reviewed the SQL Poison Wait accumulation-shape port. The evaluation logic (PoisonWaitEvaluator, the AlertEngine fire/clear/re-fire arms, the shared-constant aliasing with PostgresAlertEvaluator, and the DuckDB/Postgres SQL parity) all check out against the stated design — boundary grading, worst-first ordering, the observed-vs-empty clear rule, and the Lite/Darling SQL text parity all match their tests and each other.
- UI regression: the "retired" poison-wait ms box doesn't actually stay disabled. Both Settings windows (
Lite/Windows/SettingsWindow.xamlandDarling/PerformanceMonitor.Darling.Viewer/SettingsWindow.xaml) setAlertPoisonWaitThresholdBoxtoIsEnabled="False"specifically so operators can't tune a value the engine no longer consults. But in bothSettingsWindow.xaml.csfiles,UpdateAlertControlStates()unconditionally doesAlertPoisonWaitThresholdBox.IsEnabled = enabled(whereenabledmirrors the master "Alerts Enabled" checkbox), and that method runs both on window load and on every toggle of that checkbox. With alerts enabled — the default — the box re-enables itself as soon as the Settings window opens, silently undoing the disablement this PR added and defeating its own stated intent ("the box is disabled ... so nobody tunes a number nothing reads"). Present identically in both SKUs, so it's a shared bug rather than a parity drift, but it does need a fix (exclude that control from the loop, or force it disabled unconditionally after).
Everything else — the evaluator math, the read-adapter SQL, the clear/re-fire state machine, and the extensive test coverage — looks correct and consistent between Lite and Darling.
…gh the master-switch loop, both SKUs (review round 3)
|
Round-3 finding (the retired ms box re-enabled by |
Round-3 finding (master-switch loop re-enabling the retired ms box) remediated on both SKUs; see the PR comment and inline replies.
| window on both engines (before that, SQL Server's average ms per wait; rows from then still | ||
| render in the same unit, which is why the arm did not change). Long-running query carries | ||
| elapsed minutes. */ | ||
| "Poison Wait" => $"{value:F0} ms", |
There was a problem hiding this comment.
Non-blocking, since the PR body already discloses this under "What it does NOT do" and hands it to the coordinator — flagging for visibility since it's a real (if pre-existing-shaped) behavior change.
AlertMetricClassifier.IsCritical (a few lines up in this file) classifies "Poison Wait" by name, unconditionally, for the Lite and Darling Viewer Alert History grids. Before this PR that was accurate, because the SQL Server engine only ever fired Poison Wait presence-flat CRITICAL. After this PR, SQL Server (and PostgreSQL, unchanged) can genuinely fire Poison Wait at Warning severity (1.0–9.99 avg waiters, a real fraction of the fired range, not an edge case). The email/webhook paths correctly consult Context.SeverityOverride via AlertSeverity.ForMetric, but the History grids' IsCritical/IsWarning have no override mechanism — so a Warning-graded poison-wait row will still render red/CRITICAL in both apps' history grids, silently overstating its severity relative to what the toast/email/webhook just said for the same event.
There was a problem hiding this comment.
Agreed, and deliberately out of this lane: IsCritical/IsWarning are by-name for every metric and have no override channel, so the honest fix is a grid-side severity column (the history row already serializes Context.SeverityOverride in its context JSON on Darling; Lite's row does not carry it) — a surface change across both apps' Alert History grids, not a one-arm edit here. It has been a real mismatch for PG Warning poison fires since #2711; this PR widens the population it affects. Handed to the coordinator as a named out-of-lane finding with the mechanism above.
|
Reviewed the full diff (production code, test changes, and both Settings-window/README parity edits). Correctness. The new accumulation shape checks out: the shared Lite/Darling parity. One residual worth a second look (left as an inline comment on Security/T-SQL style. No SQL Server T-SQL touched by this PR (Darling's PostgreSQL adapter text and Lite's DuckDB text are the only SQL surfaces changed); both remain parameterized ( |
There was a problem hiding this comment.
LGTM — reviewed the accumulation-shape port for correctness (grading boundaries, worst-first sort, clear-arm observed/quiet distinction, #2704 freshness guard), Lite/Darling parity across the read adapters and both Settings windows, and SQL parameterization. One residual noted inline (AlertMetricClassifier.IsCritical's by-name Poison Wait styling in the history grids no longer matches the now-real Warning tier) but it's explicitly disclosed in the PR body as deferred, so non-blocking.
…rs, their Long-Running Query alert skips maintenance like SQL Server's does, and every one-sided alert says why it is one-sided (#3539 parity) The fleet card's deadlock dot on a PostgreSQL target read Unknown while pg_database_stats held the server's own deadlock counter, because the fleet reader took v_deadlocks (the SQL Server extended-event capture) and ServerMetricSources nulled the structural zero. Both fleet surfaces (the service's get_fleet_overview / /api/fleet reader and the viewer's Overview card and totals) now difference pg_stat_database.deadlocks per (server, database) series between consecutive samples, clamp at zero across a statistics reset, sum across databases, and band the result through the SAME deadlock_warn_per_hour / deadlock_critical_per_hour tiers as the SQL Server graph count. Never SUM(deadlocks): the column is a lifetime counter repeated in every sample. FleetDeadlockSource.PostgresTarget now means "counted from the server counter" and is a covered arm; servers_read counts it and postgres_servers is a sub-count. A PostgreSQL target whose pg_database_stats collector is silent or denied reads those arms, exactly as a SQL Server whose deadlocks collector is. The PostgreSQL Long-Running Query read gains the noise opt-outs SQL Server's has had: non-client backends (mirrors session_id > 50), the VACUUM/ANALYZE/REINDEX/CLUSTER statements by command tag (no text is stored, so the tag is the only handle), the dump/restore utilities by application_name on the shared longRunningQueryExcludeBackups switch, and excludedDatabases after the read. CREATE stays reported with its tag. The Darling README gains an engine-coverage table for every built-in alert and band, stating why Blocking Wait Time and Database File Growth are SQL-only, why the PostgreSQL blocking BAND stays Unknown (sampled evidence against tiers measured on engine-recorded reports), that Poison Wait is one shape since #3593, and that slot retention's SQL analogue (log_reuse_wait_desc) is collected only as a load-time snapshot today.
…rs, their Long-Running Query alert skips maintenance like SQL Server's does, and every one-sided alert says why it is one-sided (#3539 parity) The fleet card's deadlock dot on a PostgreSQL target read Unknown while pg_database_stats held the server's own deadlock counter, because the fleet reader took v_deadlocks (the SQL Server extended-event capture) and ServerMetricSources nulled the structural zero. Both fleet surfaces (the service's get_fleet_overview / /api/fleet reader and the viewer's Overview card and totals) now difference pg_stat_database.deadlocks per (server, database) series between consecutive samples, clamp at zero across a statistics reset, sum across databases, and band the result through the SAME deadlock_warn_per_hour / deadlock_critical_per_hour tiers as the SQL Server graph count. Never SUM(deadlocks): the column is a lifetime counter repeated in every sample. FleetDeadlockSource.PostgresTarget now means "counted from the server counter" and is a covered arm; servers_read counts it and postgres_servers is a sub-count. A PostgreSQL target whose pg_database_stats collector is silent or denied reads those arms, exactly as a SQL Server whose deadlocks collector is. The PostgreSQL Long-Running Query read gains the noise opt-outs SQL Server's has had: non-client backends (mirrors session_id > 50), the VACUUM/ANALYZE/REINDEX/CLUSTER statements by command tag (no text is stored, so the tag is the only handle), the dump/restore utilities by application_name on the shared longRunningQueryExcludeBackups switch, and excludedDatabases after the read. CREATE stays reported with its tag. The Darling README gains an engine-coverage table for every built-in alert and band, stating why Blocking Wait Time and Database File Growth are SQL-only, why the PostgreSQL blocking BAND stays Unknown (sampled evidence against tiers measured on engine-recorded reports), that Poison Wait is one shape since #3593, and that slot retention's SQL analogue (log_reuse_wait_desc) is collected only as a load-time snapshot today.
…rs, their Long-Running Query alert skips maintenance like SQL Server's does, and every one-sided alert says why it is one-sided (#3539 parity) (#3638) The fleet card's deadlock dot on a PostgreSQL target read Unknown while pg_database_stats held the server's own deadlock counter, because the fleet reader took v_deadlocks (the SQL Server extended-event capture) and ServerMetricSources nulled the structural zero. Both fleet surfaces (the service's get_fleet_overview / /api/fleet reader and the viewer's Overview card and totals) now difference pg_stat_database.deadlocks per (server, database) series between consecutive samples, clamp at zero across a statistics reset, sum across databases, and band the result through the SAME deadlock_warn_per_hour / deadlock_critical_per_hour tiers as the SQL Server graph count. Never SUM(deadlocks): the column is a lifetime counter repeated in every sample. FleetDeadlockSource.PostgresTarget now means "counted from the server counter" and is a covered arm; servers_read counts it and postgres_servers is a sub-count. A PostgreSQL target whose pg_database_stats collector is silent or denied reads those arms, exactly as a SQL Server whose deadlocks collector is. The PostgreSQL Long-Running Query read gains the noise opt-outs SQL Server's has had: non-client backends (mirrors session_id > 50), the VACUUM/ANALYZE/REINDEX/CLUSTER statements by command tag (no text is stored, so the tag is the only handle), the dump/restore utilities by application_name on the shared longRunningQueryExcludeBackups switch, and excludedDatabases after the read. CREATE stays reported with its tag. The Darling README gains an engine-coverage table for every built-in alert and band, stating why Blocking Wait Time and Database File Growth are SQL-only, why the PostgreSQL blocking BAND stays Unknown (sampled evidence against tiers measured on engine-recorded reports), that Poison Wait is one shape since #3593, and that slot retention's SQL analogue (log_reuse_wait_desc) is collected only as a load-time snapshot today.
… idiom (#3653 Dashboard mirrors) (#3658) - get_blocking / get_deadlocks / get_alert_history: page counts stop posing as totals (*_returned, truncated, limit_applied_by_reader) - #3594's class - Poison Wait: accumulated over the ten-minute window, graded on the shared PoisonWaitEvaluator bars, clear needs an observation - #3593's class - mute_analysis_finding: registered / matched_now / muted_unmatched - #3615's class - CollectorSeverity (0,0) online -> Unknown, '--' - #3635's class - cntr_value_per_second recomputed as a fraction on the read - #3540 A11 - SqlServerAnomalyDetector lab-box markers replaced with what #3616 measured (comment-only)
What does this PR do?
SQL Server's Poison Wait alert now measures accumulated starvation over a rolling ten-minute window and grades it WARNING/CRITICAL — the shape its PostgreSQL twin has fired on since #2711 — so one slow wait no longer pages and a THREADPOOL storm no longer sleeps. Partial for #3539: checklist item A4 ("SQL Poison Wait shape blindness") and the parity-table gap it names (the undocumented always-CRITICAL-vs-graded severity divergence under one shared mute key). References #3539; does not close it.
Root cause
AlertEngine.CheckPoisonWaitsAsyncasked the read adapter for the newest threewait_statsrows (withdelta_waiting_tasks > 0) and judged each row'savg_ms_per_wait >= PoisonWaitThresholdMs(500), presence-flat CRITICAL through theAlertSeveritymap. That is a statement about how long ONE wait lasted in ONE collector interval, with no volume floor. It has both failure modes at once:Meanwhile "Poison Wait" is ONE alert name on both engines — one
PoisonWaitEnabledswitch, one mute key, one history filter — and the PostgreSQL side (#2711) already fired on TOTAL accumulated wait over a ten-minute window normalized to "average backends continuously stuck", graded Warning at 1.0 / Critical at 10.0, fleet-calibrated ~160× above the worst healthy baseline. A mute rule written against one meaning silently governed the other, and the divergence was documented nowhere.What the fleet said before a line changed
One production store class, 43 SQL Server primaries, 4 days of
wait_stats, 24,757 server-ten-minute buckets (sanitized figures; the measurement document lives with the lane):avg >= 500trigger fired on no row in four days — no observed false positive, and nothing poison happened either. But its near-misses show both wrong shapes: THREADPOOL rows at 247.5 ms avg over 2 tasks and 163 ms over 1 task (the one-slow-wait class — a 600 ms sibling would have paged), and the largest THREADPOOL row at 703 tasks × 8.2 ms (the storm shape, invisible to an average). One server's daily compile burst (8 tasks, ~3 s, 356–394 ms avg) sat just under the old bar.delta_wait_time_ms > 0anddelta_waiting_tasks = 0(304, 151, 149, 146 ms in one interval) — a task still waiting across the interval boundary. The old read's task filter dropped them; accumulation counts them.Nothing measured would fire the new shape, which is the point: these waits are near-zero when healthy, and a full task pinned for ten straight minutes is categorically not that.
What changed, and why this shape
One shared home for the bars. New
PoisonWaitEvaluator(Alerting) ownsWindowMinutes = 10,WarningAvgWaiters = 1.0,CriticalAvgWaiters = 10.0, the sharedGrade, and the SQL Server evaluation.PostgresAlertEvaluator's three poison constants are now aliases of those (= PoisonWaitEvaluator.X) under their old names, so the PG read adapter, the PG host and the PG tests keep compiling untouched — and a test reads the source to pin that they are aliases, not equal literals (two equal literals are exactly the state that drifts). Both engines' margins are documented on the shared constants. No engine-specific reason to diverge was found; the rationale for sharing (D9-style: one name under one mute key means one thing) is in the class comment, with the instruction that a future SQL-specific bar gets its own documented constant beside these, not a silent local number.The read is a dumb window sum.
IAlertReadAdapter.GetPoisonWaitDeltasAsync(serverKey, thresholdMs)→GetPoisonWaitAccumulationAsync(serverKey, windowMinutes)returningPoisonWaitAccumulation(WaitType, AccumulatedWaitMs, AccumulatedWaits, ObservedIntervals, NewestCollectionTime)— the SQL Server twin ofPostgresPoisonWaitAlertInfo. Both SKUs' SQL:SUM(delta_wait_time_ms),SUM(delta_waiting_tasks),COUNT(*),MAX(collection_time)per poison wait type overcollection_time >= now − window, nodelta_waiting_tasks > 0filter (that wait is evidence), noLIMIT(a limit on a sum is an undercount), no threshold (the engine grades; and it needs the under-bar rows — see the clear arm). The window is passed by the engine rather than read from the constant in the adapter so the read's cutoff and the evaluator's denominator are one number by construction. (0, 0) rows — the delta calculator's "no delta is knowable" marker, indistinguishable inwait_statsfrom an idle interval — contribute nothing to the sum, and the denominator is the window's wall clock, not a row count, so a fabricated zero neither inflates nor deflates the figure and is never itself a claim of quiet. Deliberately no interval handling: the V128 keystone lane ownssample_interval_secondsand rebases onto this text.Read cost under the alert pass's 10-second deadline. The alert pass on the largest store has to survive the hourly interval CAGG refresh (4–7 minutes, #3597): a read that is sub-second by shape survives, a 2-second one dies. This read is cheap by SHAPE, not by any index the planner may or may not choose — no claim is made about the
(server_id, collection_time)composite, which the planner has refused on a sibling table with the same interleaved write pattern (#3573). The WHERE is predicate-identical to the retired read's (oneserver_id, the threewait_typeliterals,collection_time >= now − 10 min), so it touches exactly the rows the old read touched and aggregates them instead of ordering them for aLIMIT 3: at the collector's one-minute cadence, at most ~10 collections × 3 types ≈ 30 rows per server, all in the last ten minutes of the current one-day chunk (the uncompressed head; chunk exclusion keeps compressed history out of the plan). The PostgreSQL twin has run this exact aggregate shape overpg_wait_statsunder the same deadline since #2711. Stated in the SQL's own doc comment.The engine grades and fires ONE alert per server, as before. Per wait type, worst-first (severity, then accumulated ms); the alert's severity is the worst type's, carried on BOTH
AlertOutcome.SeverityandAlertContext.SeverityOverride(channel builders read the context — the #1136 low-disk path; Darling's deliverer folds the outcome's in only when the context has none — #2090; setting both makes both SKUs agree by construction). Mute keys on the worst wait type (Lite's documented limitation, unchanged). Detail items per firing type carry accumulated seconds, avg tasks stuck, waits completed, the tier and a per-type remedy (SqlServerRemedyFor— the sibling of PG'sPoisonWaitRemedyFor, expanding the README's own one-line characterizations). Numerics are the worst type's accumulated ms against the breached bar in ms — the PG twin's exact numeric pair and the unit the history formatter already renders this metric in. Message shape mirrors PG clause-for-clause with the engine's noun (task(s)vsbackend(s)):[THREADPOOL] 600s of wait accumulated in the last 10 minutes across 300,000 waits — on average 1.0 task(s) continuously stuck. Not one alert per wait type like the PG host: changing the per-server delivery contract (metric-level cooldown fallback, no incidents, mute-on-worst) is a separate decision from changing what the number means.The clear arm requires an observation. A standing alert clears when the window holds at least one poison-type row AND no type is over the bar — which, because the sum is rolling, happens only once enough wait has aged out, up to a full window late (the PG twin accepts the same lateness). An empty read holds the flag where it is: no
wait_statsrows for any poison type in ten minutes is the collector not delivering, not the server going quiet (THREADPOOL has lifetime wait on any server up long enough to fire this, so its row lands every cycle). The retired shape announced "Poison Waits Cleared" on that same silence. This is the #3282 rule for a CPU reading that stops arriving, applied here; it is NOT theAlertPersistenceGate— the rolling window is already this alert's persistence in both directions (600 s must accumulate to fire; the sum must fall back under the bar to clear), and A5 (gate rollout) is a separate #3539 item this PR does not touch.The #2704 re-fire guard stays, keyed on the newest collection_time of the FIRING types (a quiet sibling's newer row is not a new observation of the type over the bar).
PoisonWaitThresholdMsretired honestly. It is persisted on both SKUs (Lite settings JSON; Darling'sdarling.jsonand the store control-plane row) and surfaced by both Settings windows andget_alert_settings. It stays onIAlertEngineSettings, stays readable/writable/reported, and is no longer consulted — the interface member's doc says why (no migration rung for a value nothing evaluates; a per-wait average has no honest meaning to repurpose it to). Pass-throughs onDarlingAlertSettings/AppAlertEngineSettings/DarlingConfigcarry a one-line pointer. Both READMEs say so; the README's alert table (which claimed "100 ms avg" — a stale default on top of a stale shape) now states the windowed bars, the clear rule, and that SQL and PG share the shape.AlertSeveritymap arm for "Poison Wait" keeps CRITICAL with a new comment: every live fire now carries its grade on the context, so the arm styles only override-less renders — alert-history replays of rows written BEFORE this change, every one of which WAS a presence-flat CRITICAL fire. CRITICAL is therefore the faithful replay, not a default.Alternatives rejected
PostgresAlertEvaluator; the shared class is the honest home, with aliases so nothing PG-side moves.AlertPersistenceGatefor the clear. The window is the persistence; gating it would count breaches in collector cycles and clears in sweeps (the units mismatch the field comment already names) on top of a measure that already holds.Blast radius
IAlertReadAdaptermember replaced (no default implementations by rule): compiler named every implementer and fake —DarlingAlertReadAdapter,LiteAlertReadAdapter(+LocalDataService.WaitStats, where Lite's DuckDB text lives by the adapter's own "engine stays free of DuckDB" convention),AlertEngineTests(two fakes),DarlingSelfAlertTests,LiteAlertForwardingTests. All updated.PoisonWaitDeltaandAlertContextBuilders.BuildPoisonWaitContextare kept — the deprecated Dashboard's own loop uses them;PoisonWaitDelta's comment now says the shared engine no longer does.AlertReadFailureSurfaceTestscensus:catch (Exception)count inAlertEngine.csunchanged (20 total / 14 counted), every await in the poison try block still followed byreadClock.Restart().DarlingAlertReadAdapterTestsnow seeds the measured shapes (703 × 8 ms row, time-with-no-tasks row, (0, 0) row, storm row, out-of-window row, sub-bar sibling) and asserts the sums;DarlingAlertingTestsplants 36,000 × 20 ms = 720 s (the old shape could not fire on a 20 ms average).What it does NOT do
CHANGELOG.md(entry handed to the coordinator).sample_interval_secondshandling (V128 lane).get_alert_settings.poison_wait.threshold_ms(still reports the unconsulted value),AlertMetricClassifier.IsCritical's by-name "Poison" emphasis in the history grids, the deprecated Dashboard's own avg-based loop, or the PG host's clear-on-empty arm — outside this lane's file boundary and listed for the coordinator. (The Settings-window labels and save-preview strings on Lite and the Viewer WERE fixed after review — a label this change falsified is part of this change; see the review disposition comment.)Test plan
New / changed tests — all compile-verified here (
Darling.Tests,Lite.Tests: 0 warnings) and first EXECUTED in CI (net10.0-windows). Before trusting them, the engine scenarios were run on this machine against the realAlertEnginewithAlertEngineTests' own fakes in a throwaway console under a hostile (de-DE) culture: every assertion below passed; and both SQL texts were executed against a throwaway PG18 + TimescaleDB 2.28.1 container (desktop-linux, torn down) and an in-memory DuckDB 1.5.5 with the realDuckDBParameterbinding — both returnedTHREADPOOL 600083 / 30403 / 4andRESOURCE_SEMAPHORE 3154 / 8 / 1on the seeded shapes, with the out-of-window and other-server rows excluded.Darling.Tests/AlertEngineTests(poison-wait section, rewritten):PoisonWait_FiresOnAccumulatedStarvation_NotPerWaitAverage— 300,000 × 2 ms = 600 s fires Warning; exact CurrentValue/ThresholdValue/ShortMessage; numerics 600,000 / 600,000; severity on outcome AND context; one detail with remedy; no incidents; window asked = 10.PoisonWait_OneSlowWait_AndEveryMeasuredFleetBucket_StaySilent— 1 × 600 ms, 5,795, 5,779 (703 tasks), 3,154 (compile burst), 599,999: all silent.PoisonWait_GradesCritical_AtTenTasksContinuouslyStuck— 6,000,000 ms → Critical, threshold text names the 10-task bar.PoisonWait_JudgesEachWaitTypeIndependently_WorstFirst_MutesOnTheWorst— Critical RS leads Warning TP (5.9M), sub-bar third type absent from value and details; mute asked for the worst.PoisonWait_HoldsOnAnEmptyRead_AndClearsOnlyOnAnObservedQuietWindow— empty read: no fire, no clear; zero-sum observed rows: Cleared with the windowed message; Cleared is an edge.PoisonWait_ClearsOnASubBarWindow_RegardlessOfTheRetiredKnob— knob = 1 ms changes nothing.PoisonWait_DoesNotRefire_OnTheSameCollectionTime_EvenAfterCooldownElapses(Fix Poison Wait alert re-firing on an unrefreshed wait_stats row #2704, re-shaped) andPoisonWait_FreshnessFollowsTheFiringTypes_NotAQuietSibling.PoisonWait_Suppressed_TracksTheConditionWithoutDelivering.Darling.Tests/PostgresAlertEvaluatorTests(new SQL-port section): constants are ONE definition (value + source-alias pin with positive control); both engines grade identically at 0 / 599,999 / 600,000 / 5,999,999 / 6,000,000 with the same numeric pair; storm fires / one slow wait does not; fleet-quiet pin; (0, 0) is no evidence; worst-first ordering; message shape mirrors PG; remedy per type.Darling.Tests/DarlingAlertReadAdapterTests:PoisonWaitsSql_IsAWindowAccumulation_NotTheNewestDeltas(absences: task filter, LIMIT, threshold,sample_interval; presences: the four aggregates, GROUP BY; wait-type list equals the evaluator's); the live E2E re-planted as above.Lite.Tests:PoisonWait_ToastBody_UsesTheWorstWait_GradedByAccumulation_NotTheRetiredKnob(knob = 999, which under the old shape silenced both rows);PoisonWaitAccumulationSql_IsTheDarlingTextInDuckDbDialect(clause-for-clause parity read from both sources viaParitySource, with the only permitted dialect differences named); reflection pins moved to the new member.Builds (all zero warnings):
PerformanceMonitor.Alerting,PerformanceMonitor.Darling.Service,Lite(-p:EnableWindowsTargeting=true),Darling.Tests,Lite.Tests.Which component(s) does this affect?
How was this tested?
See the test plan: compile-verified on macOS, evaluator/engine scenarios executed locally against the real engine, SQL executed against throwaway PG18+Timescale and DuckDB. No SQL Server was touched; the read runs against the store, not a target.
Checklist
dotnet build -c Debug)