You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Poison Wait alert re-fires on the same wait_stats row when the collector lags the cooldown #2703
The "Poison Wait" alert (`THREADPOOL` / `RESOURCE_SEMAPHORE` / `RESOURCE_SEMAPHORE_QUERY_COMPILE`) fired byte-identical duplicate alerts on live production monitoring, minutes apart, on the same server:
`phi-01`: two alerts at 05:54:51 and 06:01:27 UTC, both `RESOURCE_SEMAPHORE_QUERY_COMPILE`, avg 850.7 ms/wait, delta wait 113,997 ms, delta tasks 134 — identical to the decimal.
Same pattern earlier the same night across three servers (`multi25`, `multi-22`, `phi-01`) at 03:25/03:30, identical values twice ~5 minutes apart before clearing.
Root cause
`AlertEngine.CheckPoisonWaitsAsync` (`PerformanceMonitor.Alerting/AlertEngine.cs`) gates re-firing purely on wall-clock cooldown elapsed (`CooldownElapsed`, a fixed `TimeSpan` compared against the last-fired timestamp). It has no way to know whether the underlying data has actually changed since the last fire.
`IAlertReadAdapter.GetPoisonWaitDeltasAsync` (both the Darling/Postgres and Lite/DuckDB implementations) always asks "what are the newest poison-wait `wait_stats` rows within the last 10 minutes" — `ORDER BY collection_time DESC LIMIT 3`. If the collector's delivered cadence lags the alert cooldown (already documented fleet behavior at load — delivered sweep cadence can run well behind the configured cadence), the SAME still-unrefreshed row is the answer to that question on two consecutive sweeps. The cooldown elapsing is not proof a new observation exists, so the engine fires again on data it already reported.
This is different from CPU's alert, which is legitimately level-triggered — CPU is resampled live every sweep, so an identical reading really is a fresh independent observation of a standing condition. A wait_stats delta is one collector cycle's computation; reading it twice is the same event surfacing twice.
Fix
Added `PoisonWaitDelta.CollectionTime` (the row's own `collection_time`) and threaded it through both read-adapter implementations. `AlertEngine` now tracks the collection_time it last actually fired on per server, and only re-fires when BOTH the cooldown has elapsed AND the newest triggered row's collection_time is newer than the one last fired on. A cooldown-elapsed re-ask against an unrefreshed collection_time now waits for the next sweep instead of re-firing.
Activity
erikdarlingdata commentedon Sep 1, 2026
Fixed by #2704 (unrefreshed wait_stats row guard). Merged to
dev; ships in the next release.Fix Poison Wait alert re-firing on an unrefreshed wait_stats row (eri…
Fix Collector Cost Regression alert re-firing on an unrefreshed colle…