Skip to content

Poison Wait alert re-fires on the same wait_stats row when the collector lags the cooldown #2703

Closed

Description

@erikdarlingdata

What's happening

The "Poison Wait" alert (`THREADPOOL` / `RESOURCE_SEMAPHORE` / `RESOURCE_SEMAPHORE_QUERY_COMPILE`) fired byte-identical duplicate alerts on live production monitoring, minutes apart, on the same server:

  • `phi-01`: two alerts at 05:54:51 and 06:01:27 UTC, both `RESOURCE_SEMAPHORE_QUERY_COMPILE`, avg 850.7 ms/wait, delta wait 113,997 ms, delta tasks 134 — identical to the decimal.
  • Same pattern earlier the same night across three servers (`multi25`, `multi-22`, `phi-01`) at 03:25/03:30, identical values twice ~5 minutes apart before clearing.

Root cause

`AlertEngine.CheckPoisonWaitsAsync` (`PerformanceMonitor.Alerting/AlertEngine.cs`) gates re-firing purely on wall-clock cooldown elapsed (`CooldownElapsed`, a fixed `TimeSpan` compared against the last-fired timestamp). It has no way to know whether the underlying data has actually changed since the last fire.

`IAlertReadAdapter.GetPoisonWaitDeltasAsync` (both the Darling/Postgres and Lite/DuckDB implementations) always asks "what are the newest poison-wait `wait_stats` rows within the last 10 minutes" — `ORDER BY collection_time DESC LIMIT 3`. If the collector's delivered cadence lags the alert cooldown (already documented fleet behavior at load — delivered sweep cadence can run well behind the configured cadence), the SAME still-unrefreshed row is the answer to that question on two consecutive sweeps. The cooldown elapsing is not proof a new observation exists, so the engine fires again on data it already reported.

This is different from CPU's alert, which is legitimately level-triggered — CPU is resampled live every sweep, so an identical reading really is a fresh independent observation of a standing condition. A wait_stats delta is one collector cycle's computation; reading it twice is the same event surfacing twice.

Fix

Added `PoisonWaitDelta.CollectionTime` (the row's own `collection_time`) and threaded it through both read-adapter implementations. `AlertEngine` now tracks the collection_time it last actually fired on per server, and only re-fires when BOTH the cooldown has elapsed AND the newest triggered row's collection_time is newer than the one last fired on. A cooldown-elapsed re-ask against an unrefreshed collection_time now waits for the next sweep instead of re-firing.

Activity

  1. erikdarlingdata commented on Sep 1, 2026

    @erikdarlingdata
    OwnerAuthor

    Fixed by #2704 (unrefreshed wait_stats row guard). Merged to dev; ships in the next release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions