Repository navigation
The wait-profile detector still fires on the window peak alone: its inline gate has no mean, unlike every detector #3724 pair-gated (Q1 residue) #3741
Description
The lie
#3724 (Q1 of #3653, first slice) made every baseline detector fire only when the window's peak AND the window's mean both clear the cutoff — AnomalyGate.EvaluateZScore grew the pair-gate overloads (PerformanceMonitor.Analysis/AnomalyGate.cs:132 onward, "the PAIR gate on the classical frame — fires only when the window peak …"), every window SQL in both detector bodies grew an AVG(...) beside its MAX(...), and a structural pin in DarlingAnomalyBaselineTests asserts that "every z-score call in BOTH detector bodies passes peak*, avg* and no peak-only call survives".
One detector never went through AnomalyGate and so never got the mean: the wait-profile detector. Its window SQL computes a peak rate and nothing else —
SELECT MAX(CASE WHEN interval_sec > 0 THEN total_wait_ms / interval_sec END) AS peak_ms_per_sec,
SUM(total_wait_ms) AS total_wait_ms,
COUNT(*) FILTER (WHERE interval_sec IS NOT NULL) AS sample_count
FROM per_collection
(Darling/PerformanceMonitor.Darling.Analysis/PgAnomalyDetector.cs, WaitRateWindowSql, ~:179-199; Lite twin Lite/Analysis/AnomalyDetector.cs:405) — and its gate is inline, on the peak alone:
if (modifiedZ < HeavyTailModifiedZThreshold || peakRate < WaitProfileFallbackMsPerSec) return;
(PgAnomalyDetector.DetectWaitAnomalies, :572; Lite AnomalyDetector.DetectWaitAnomalies, :439), with the classical-ratio and no-baseline arms beneath it likewise testing peakRate only. So the A8 shape #3724 removed everywhere else — one hot sample in an otherwise quiet window fires the anomaly — is intact for wait profile, the one metric the fleet measured as heavy-tailed by nature. #3724's body says so ("Does not pair-gate the wait-profile detector … Same A8 shape, separate structure — flagged … for a ruling") and its structural pin deliberately excludes this detector.
What is verified at source (origin/dev 98d52fe)
PgAnomalyDetector.csWaitRateWindowSqlselectsMAX(...) AS peak_ms_per_sec; noAVG.:572gate onmodifiedZ(of the peak) andpeakRate.Lite/Analysis/AnomalyDetector.cs:405same SQL shape;:439same gate;:450the no-baseline arm onpeakRatealone.AnomalyGate.cs:132-the pair-gate overload the other detectors call; the wait-profile detector does not referenceAnomalyGate.
Why it needs a ruling and not just a patch
The wait-profile gate is a modified z on a heavy-tailed metric with a 5.0 cutoff (HeavyTailModifiedZThreshold), measured in #1743 to strictly contain the ratio trigger's catches at every cutoff. A mean-rate gate on a heavy-tailed distribution behaves differently from one on CPU or sessions: the mean of a spiky wait window is dragged by the spike it is meant to discount. The question for Erik: (a) pair-gate on the window mean ms/sec under the same 5.0 robust cutoff, (b) pair-gate on a robust window center (median ms/sec across the window's collections — the SQL already groups per collection), or (c) leave the peak-only gate and route the detector through AnomalyGate with the pair arms disabled so the structural pin can include it. The engine's own precedent is (a) for every other detector; the honest answer may be (b).
The fix shape (after the ruling)
- Add the second aggregate to
WaitRateWindowSqlon both SKUs (AVG(...)or apercentile_cont(0.5)/median()twin), read it beside the peak, and gate on both throughAnomalyGate— the inlineifcollapses into oneEvaluateZScorecall withHeavyTailModifiedZThresholdas its cutoff andWaitProfileFallbackMsPerSecas its floor, so the structural pin ("no peak-only call survives") can drop its exclusion. - Metadata:
current_ms_per_secstays the peak; the new center rides beside it soFactAdvice's wait-profile prose can name both, as The anomaly gate judges the window peak AND the window mean, so one hot sample stops reading as an anomaly and a longer window stops buying more of them; I/O reads the pair like its siblings (Partial #3653 — item 1, Q1) #3724 did for the others. - Fixtures on the The anomaly gate judges the window peak AND the window mean, so one hot sample stops reading as an anomaly and a longer window stops buying more of them; I/O reads the pair like its siblings (Partial #3653 — item 1, Q1) #3724 pattern: one hot collection in a quiet window → no fire; both clear → fire; the no-baseline fallback bar still on the peak.
Activity
The wait-profile detector judges the window peak AND the window mean:…
erikdarlingdata commentedon Sep 20, 2026
Closed by #3773 (merged to dev).