Skip to content

The wait-profile detector still fires on the window peak alone: its inline gate has no mean, unlike every detector #3724 pair-gated (Q1 residue) #3741

Closed

Description

@erikdarlingdata

The lie

#3724 (Q1 of #3653, first slice) made every baseline detector fire only when the window's peak AND the window's mean both clear the cutoff — AnomalyGate.EvaluateZScore grew the pair-gate overloads (PerformanceMonitor.Analysis/AnomalyGate.cs:132 onward, "the PAIR gate on the classical frame — fires only when the window peak …"), every window SQL in both detector bodies grew an AVG(...) beside its MAX(...), and a structural pin in DarlingAnomalyBaselineTests asserts that "every z-score call in BOTH detector bodies passes peak*, avg* and no peak-only call survives".

One detector never went through AnomalyGate and so never got the mean: the wait-profile detector. Its window SQL computes a peak rate and nothing else —

SELECT MAX(CASE WHEN interval_sec > 0 THEN total_wait_ms / interval_sec END) AS peak_ms_per_sec,
       SUM(total_wait_ms) AS total_wait_ms,
       COUNT(*) FILTER (WHERE interval_sec IS NOT NULL) AS sample_count
FROM per_collection

(Darling/PerformanceMonitor.Darling.Analysis/PgAnomalyDetector.cs, WaitRateWindowSql, ~:179-199; Lite twin Lite/Analysis/AnomalyDetector.cs:405) — and its gate is inline, on the peak alone:

if (modifiedZ < HeavyTailModifiedZThreshold || peakRate < WaitProfileFallbackMsPerSec) return;

(PgAnomalyDetector.DetectWaitAnomalies, :572; Lite AnomalyDetector.DetectWaitAnomalies, :439), with the classical-ratio and no-baseline arms beneath it likewise testing peakRate only. So the A8 shape #3724 removed everywhere else — one hot sample in an otherwise quiet window fires the anomaly — is intact for wait profile, the one metric the fleet measured as heavy-tailed by nature. #3724's body says so ("Does not pair-gate the wait-profile detector … Same A8 shape, separate structure — flagged … for a ruling") and its structural pin deliberately excludes this detector.

What is verified at source (origin/dev 98d52fe)

  • PgAnomalyDetector.cs WaitRateWindowSql selects MAX(...) AS peak_ms_per_sec; no AVG. :572 gate on modifiedZ (of the peak) and peakRate.
  • Lite/Analysis/AnomalyDetector.cs:405 same SQL shape; :439 same gate; :450 the no-baseline arm on peakRate alone.
  • AnomalyGate.cs:132- the pair-gate overload the other detectors call; the wait-profile detector does not reference AnomalyGate.

Why it needs a ruling and not just a patch

The wait-profile gate is a modified z on a heavy-tailed metric with a 5.0 cutoff (HeavyTailModifiedZThreshold), measured in #1743 to strictly contain the ratio trigger's catches at every cutoff. A mean-rate gate on a heavy-tailed distribution behaves differently from one on CPU or sessions: the mean of a spiky wait window is dragged by the spike it is meant to discount. The question for Erik: (a) pair-gate on the window mean ms/sec under the same 5.0 robust cutoff, (b) pair-gate on a robust window center (median ms/sec across the window's collections — the SQL already groups per collection), or (c) leave the peak-only gate and route the detector through AnomalyGate with the pair arms disabled so the structural pin can include it. The engine's own precedent is (a) for every other detector; the honest answer may be (b).

The fix shape (after the ruling)

Descends from #3724 (Q1 of #3653). Partial #3653 residue.

Activity

  1. erikdarlingdata commented on Sep 20, 2026

    @erikdarlingdata
    OwnerAuthor

    Closed by #3773 (merged to dev).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions