Skip to content

The wait-profile detector judged the window peak alone: its inline gate becomes the shared AnomalyGate pair call every other baseline detector took in #3724, so one hot collection in a quiet window stops reading as a profile shift (#3741) - #3773

Merged
erikdarlingdata merged 1 commit into
devfrom
fix/3741-wait-profile-pair-gate
Sep 20, 2026

Conversation

@erikdarlingdata

Copy link
Copy Markdown
Owner

Descends from #3724 (Q1 of #3653, first slice) — this is its last leg.

The lie

#3724 made every baseline detector fire only when the window's peak AND the window's mean both clear the cutoff: AnomalyGate.EvaluateZScore grew its pair overloads, every window SQL in both detector bodies grew an AVG(...) beside its MAX(...), and a structural pin asserts no peak-only call survives. One detector never went through AnomalyGate and so never got the mean: the wait profile. Its window read computed a peak ms/sec and nothing else —

SELECT MAX(CASE WHEN interval_sec > 0 THEN total_wait_ms / interval_sec END) AS peak_ms_per_sec, ...

— and its gate was inline, on the peak alone: if (modifiedZ < HeavyTailModifiedZThreshold || peakRate < WaitProfileFallbackMsPerSec) return; (PgAnomalyDetector.DetectWaitAnomalies, Lite twin AnomalyDetector.DetectWaitAnomalies), with the classical-ratio and no-baseline arms beneath it likewise on peakRate. So the A8 shape #3724 removed everywhere else — one hot collection in an otherwise quiet window fires the anomaly, and a longer window buys more of them — was intact for the one metric the fleet measured as heavy-tailed by nature. #3724's own body flagged it for a ruling; its pin deliberately excluded the detector.

The ruling

Option (a) from #3741: pair-gate on the window mean ms/sec under the same robust cutoff (HeavyTailModifiedZThreshold = 5.0) — the engine's precedent for every other detector. Not the median, not peak-only-through-the-gate.

Mechanism

The window read — PgAnomalyDetector.WaitRateWindowSql and Lite's inline twin, verbatim on both SKUs: AVG(CASE WHEN interval_sec > 0 THEN total_wait_ms / interval_sec END) AS avg_ms_per_sec beside the MAX, same arm, same guard. A NULL-interval collection (a restart, per #3540) contributes NULL to both aggregates and both ignore it, so the peak and the mean describe the same sample set and the gate compares like with like. The column order is the reader's ordinal contract (0 peak, 1 avg, 2 total, 3 count) — pinned on the PG const and against Lite's text.

The trusted robust arm — the inline if collapses into ONE call:

var decision = AnomalyGate.EvaluateZScore(
    baseline, peakRate, avgRate,
    HeavyTailModifiedZThreshold, HeavyTailModifiedZThreshold, WaitProfileFallbackMsPerSec, WaitProfileFallbackMsPerSec, SigmaDisplayCap);
if (!decision.Fire) return;

No change to AnomalyGate was needed: the BaselineBucket pair overload already speaks the frame this arm always spoke — DecideRobustFirst judges both statistics as modified z against the bucket's median and EffectiveRobustSigma, the exact arithmetic of BaselineMath.ModifiedZScore (which still stamps the uncapped modified_z the scorer grades). Peak AND mean must both clear 5.0; the 250 ms/sec floor stays on the peak only (the gate's rule: the floor is the "trivial value" ceiling for the value the finding reports, sized for a peak). The arm's EffectiveRobustSigma > 0 guard means the gate's classical frame is unreachable here, so the same 5.0 is passed as both cutoffs and the family's one bar as both floor and bar — said in the comment rather than left for a reader to infer.

The ratio arm (a trustworthy bucket without robust statistics — #1743's "keeps the classical ratio trigger") asks the same of both ratios: peakRate / Mean AND avgRate / Mean over DefaultRatioThreshold.

The no-baseline arm stays on the peak's absolute bar alone — there is no z to trust on either statistic there, and the ruling keeps that bar where it was.

Metadata and story — current_ms_per_sec stays the peak (the value the story leads with, the ratio's numerator); avg_ms_per_sec is the window mean beside it, mean_modified_z its deviation in the same uncapped frame as modified_z. That is the wait profile's own vocabulary: the other families' deviation_sigma / mean_deviation_sigma pair is the capped one, and this fact has never carried deviation_sigma, so borrowing mean_deviation_sigma would have put a mean's deviation on the MCP baseline_context with no peak deviation beside it. FactAdvice.ComposeAnomalyWaitProfile names both: on the gate arm "the window's mean of 1606.3 ms/sec sat 9.5 robust σ above the baseline median (the detector fires only when the mean clears its bar as well, so this is the profile running heavy across the window, not one hot collection)"; on the ratio arm the mean's multiple of the baseline; on is_new the mean as plain context — the bar is on the peak alone there and the sentence must not claim a gate the detector did not apply. A pre-#3741 fact keeps its sentence verbatim. FactScorer's wait-profile arm is untouched and grades off modified_z / ratio exactly as before (pinned).

The one honest caveat, written into the comment

The mean's z is judged against the SAME per-collection median/MAD as the peak's — one centre, one dispersion, one Decide body, as #3724 did for every other family. For a heavy-tailed family that is a little LESS strict than a symmetric reading suggests: the mean of N draws converges to the distribution's mean, which sits above its median when the tail is to the right, so the mean clause's null expectation is a little above zero. It is a bias toward firing, never toward silence, and it is bounded — the mean clause can only admit a window the peak already admitted. The median-centred alternative (option b in #3741) was weighed and the ruling took the engine's precedent; the comment says so where the next reader will look.

What it does NOT do

Tests

Executed locally (mactest harnesses inside the worktree; a TimescaleDB 2.28.1-pg18 rig on desktop-linux, ladder 135):

  • Darling.Tests/DarlingAnomalyBaselineTests — 18/18 live. The The anomaly gate judges the window peak AND the window mean, so one hot sample stops reading as an anomaly and a longer window stops buying more of them; I/O reads the pair like its siblings (Partial #3653 — item 1, Q1) #3724 structural pin drops its exclusion (eight peak*, avg* pair calls per detector body, no peak-only call survives). New ungated pin WaitRateWindow_ReadsThePeakAndMeanPair_AndTheRobustArmIsTheSharedPairGate_LiteVerbatim: the four columns in the reader's ordinal order on the PG const and in Lite's text; the gate call names HeavyTailModifiedZThreshold and WaitProfileFallbackMsPerSec; the inline modifiedZ < HeavyTailModifiedZThreshold gate is gone from both SKUs; the ratio arm tests meanRatio; the no-baseline arm still tests peakRate alone. New live EndToEnd_WaitProfileDetector_PeakAndMeanPair_OneHotCollectionStaysQuiet_SustainedWindowFires_AgainstDevPostgres: three Mondays of rates cycling 100/200/300 ms/sec → Full bucket trustworthy, median 200, MAD 100 (percentile_cont hand-checked in the doc comment), robust σ 148.26; window A (fifteen at the median, one at 3200 ms/sec — peak 20.2σ over the floor, mean 1.26σ) → NO fact; window B (fifteen at 1500, one at 3200 — mean 9.5σ) → one ANOMALY_WAIT_PROFILE with current_ms_per_sec 3200, avg_ms_per_sec 1606.25, both modified z stamped, ratio the peak's. The pre-existing is_new e2e gains avg_ms_per_sec assertions. Red-first shown: with the PG gate stubbed back to peak-only, window A fires and the structural pin fails (2 red); restored, 18/18.
  • Darling.Tests/RobustBaselineTests — 23/23; new WaitProfilePair_TheStoryNamesTheMeanBesideThePeak_AndTheScorerStillGradesThePeaks (gate-arm prose with the robust σ, ratio-arm prose with the multiple, is_new prose without the "fires only when" claim, legacy sentence verbatim, scorer grade identical before/after the pair).
  • Lite.Tests/AnomalyDetectorTests — 28/28 through the real DuckDB detector; 3 new: one hot collection in a quiet window stays quiet (with the red-first arithmetic asserted in-test); sustained window fires with peak/mean/both z stamped; no-baseline fallback still fires on the peak alone with a mean under the bar. SeedWaitStatAsync gains an optional stored sampleIntervalSeconds (null default, every existing caller unchanged) so the fixtures rate deterministically on the v60 column. Red-first shown: with the Lite gate stubbed back to peak-only, exactly the hot-collection fixture fails; restored, 28/28.
  • Census: RepoFileAdoptionTests 2/2, StragglerCommandTimeoutTests 48/48, LiveCleanupConversionRatchetTests 15/15, MeasurementContractCensusTests 7/7 (the new AVG(CASE WHEN interval_sec > 0 …) arm carries the guarded opener the census demands), McpPageContractTests 43/43, FleetIdentifierScrubTests 2/2, DocCommentHygieneTests 76/77 (the one red is the documented harness artifact — the throwaway csproj in the tree). Live DarlingAnalysisPipelineTests 11/11 and AnalysisAsOfAnchorTests 3/3 (the is_new arm through the full pipeline).
  • Builds: PerformanceMonitor.Analysis, Darling.Analysis, Lite, Darling.Tests, Lite.Tests — -c Release -p:EnableWindowsTargeting=true, 0 warnings, 0 errors.

CI-first: nothing — every touched path executed here.

Closes #3741

… its inline peak-only gate becomes the shared AnomalyGate pair call every other baseline detector took in #3724, so one hot collection in a quiet window stops reading as a profile shift (#3741)

WaitRateWindowSql on both SKUs reads AVG beside MAX over the same interval-guarded
arm; the trusted robust arm is one AnomalyGate.EvaluateZScore pair call at the
heavy-tail 5.0 cutoff with the 250 ms/sec floor on the peak; the ratio arm asks the
same of both ratios; the no-baseline arm stays on the peak's absolute bar by ruling.
current_ms_per_sec stays the peak; avg_ms_per_sec and mean_modified_z ride beside
it, and the wait-profile story names both. Structural pin drops its exclusion (eight
pair calls per detector body), live PG and DuckDB fixtures on the #3724 pattern.

Closes #3741
@erikdarlingdata
erikdarlingdata enabled auto-merge (squash) September 20, 2026 01:59
@erikdarlingdata
erikdarlingdata merged commit b08ade2 into dev Sep 20, 2026
7 of 8 checks passed
@erikdarlingdata
erikdarlingdata deleted the fix/3741-wait-profile-pair-gate branch September 20, 2026 02:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant