Repository navigation
The wait-profile detector judged the window peak alone: its inline gate becomes the shared AnomalyGate pair call every other baseline detector took in #3724, so one hot collection in a quiet window stops reading as a profile shift (#3741) - #3773
Merged
Merged
Conversation
… its inline peak-only gate becomes the shared AnomalyGate pair call every other baseline detector took in #3724, so one hot collection in a quiet window stops reading as a profile shift (#3741) WaitRateWindowSql on both SKUs reads AVG beside MAX over the same interval-guarded arm; the trusted robust arm is one AnomalyGate.EvaluateZScore pair call at the heavy-tail 5.0 cutoff with the 250 ms/sec floor on the peak; the ratio arm asks the same of both ratios; the no-baseline arm stays on the peak's absolute bar by ruling. current_ms_per_sec stays the peak; avg_ms_per_sec and mean_modified_z ride beside it, and the wait-profile story names both. Structural pin drops its exclusion (eight pair calls per detector body), live PG and DuckDB fixtures on the #3724 pattern. Closes #3741
erikdarlingdata
enabled auto-merge (squash)
September 20, 2026 01:59
This was referenced Sep 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Descends from #3724 (Q1 of #3653, first slice) — this is its last leg.
The lie
#3724 made every baseline detector fire only when the window's peak AND the window's mean both clear the cutoff:
AnomalyGate.EvaluateZScoregrew its pair overloads, every window SQL in both detector bodies grew anAVG(...)beside itsMAX(...), and a structural pin asserts no peak-only call survives. One detector never went throughAnomalyGateand so never got the mean: the wait profile. Its window read computed a peak ms/sec and nothing else —— and its gate was inline, on the peak alone:
if (modifiedZ < HeavyTailModifiedZThreshold || peakRate < WaitProfileFallbackMsPerSec) return;(PgAnomalyDetector.DetectWaitAnomalies, Lite twinAnomalyDetector.DetectWaitAnomalies), with the classical-ratio and no-baseline arms beneath it likewise onpeakRate. So the A8 shape #3724 removed everywhere else — one hot collection in an otherwise quiet window fires the anomaly, and a longer window buys more of them — was intact for the one metric the fleet measured as heavy-tailed by nature. #3724's own body flagged it for a ruling; its pin deliberately excluded the detector.The ruling
Option (a) from #3741: pair-gate on the window mean ms/sec under the same robust cutoff (
HeavyTailModifiedZThreshold= 5.0) — the engine's precedent for every other detector. Not the median, not peak-only-through-the-gate.Mechanism
The window read —
PgAnomalyDetector.WaitRateWindowSqland Lite's inline twin, verbatim on both SKUs:AVG(CASE WHEN interval_sec > 0 THEN total_wait_ms / interval_sec END) AS avg_ms_per_secbeside theMAX, same arm, same guard. A NULL-interval collection (a restart, per #3540) contributes NULL to both aggregates and both ignore it, so the peak and the mean describe the same sample set and the gate compares like with like. The column order is the reader's ordinal contract (0 peak, 1 avg, 2 total, 3 count) — pinned on the PG const and against Lite's text.The trusted robust arm — the inline
ifcollapses into ONE call:No change to
AnomalyGatewas needed: theBaselineBucketpair overload already speaks the frame this arm always spoke —DecideRobustFirstjudges both statistics as modified z against the bucket's median andEffectiveRobustSigma, the exact arithmetic ofBaselineMath.ModifiedZScore(which still stamps the uncappedmodified_zthe scorer grades). Peak AND mean must both clear 5.0; the 250 ms/sec floor stays on the peak only (the gate's rule: the floor is the "trivial value" ceiling for the value the finding reports, sized for a peak). The arm'sEffectiveRobustSigma > 0guard means the gate's classical frame is unreachable here, so the same 5.0 is passed as both cutoffs and the family's one bar as both floor and bar — said in the comment rather than left for a reader to infer.The ratio arm (a trustworthy bucket without robust statistics — #1743's "keeps the classical ratio trigger") asks the same of both ratios:
peakRate / MeanANDavgRate / MeanoverDefaultRatioThreshold.The no-baseline arm stays on the peak's absolute bar alone — there is no z to trust on either statistic there, and the ruling keeps that bar where it was.
Metadata and story —
current_ms_per_secstays the peak (the value the story leads with, the ratio's numerator);avg_ms_per_secis the window mean beside it,mean_modified_zits deviation in the same uncapped frame asmodified_z. That is the wait profile's own vocabulary: the other families'deviation_sigma/mean_deviation_sigmapair is the capped one, and this fact has never carrieddeviation_sigma, so borrowingmean_deviation_sigmawould have put a mean's deviation on the MCPbaseline_contextwith no peak deviation beside it.FactAdvice.ComposeAnomalyWaitProfilenames both: on the gate arm "the window's mean of 1606.3 ms/sec sat 9.5 robust σ above the baseline median (the detector fires only when the mean clears its bar as well, so this is the profile running heavy across the window, not one hot collection)"; on the ratio arm the mean's multiple of the baseline; on is_new the mean as plain context — the bar is on the peak alone there and the sentence must not claim a gate the detector did not apply. A pre-#3741 fact keeps its sentence verbatim.FactScorer's wait-profile arm is untouched and grades offmodified_z/ratioexactly as before (pinned).The one honest caveat, written into the comment
The mean's z is judged against the SAME per-collection median/MAD as the peak's — one centre, one dispersion, one
Decidebody, as #3724 did for every other family. For a heavy-tailed family that is a little LESS strict than a symmetric reading suggests: the mean of N draws converges to the distribution's mean, which sits above its median when the tail is to the right, so the mean clause's null expectation is a little above zero. It is a bias toward firing, never toward silence, and it is bounded — the mean clause can only admit a window the peak already admitted. The median-centred alternative (option b in #3741) was weighed and the ruling took the engine's precedent; the comment says so where the next reader will look.What it does NOT do
AnomalyGate,AnomalyThresholds,FactScorer,BaselineMath,BaselineContextFormatter, or anyPgTarget*.cs. No cutoff, floor or bar changes.deprecated/Dashboard/Analysis/SqlServerAnomalyDetector.cs), whose wait detector keeps the peak-only inline gate — bug-fix-only, as The anomaly gate judges the window peak AND the window mean, so one hot sample stops reading as an anomaly and a longer window stops buying more of them; I/O reads the pair like its siblings (Partial #3653 — item 1, Q1) #3724 left it.Tests
Executed locally (mactest harnesses inside the worktree; a TimescaleDB 2.28.1-pg18 rig on
desktop-linux, ladder 135):Darling.Tests/DarlingAnomalyBaselineTests— 18/18 live. The The anomaly gate judges the window peak AND the window mean, so one hot sample stops reading as an anomaly and a longer window stops buying more of them; I/O reads the pair like its siblings (Partial #3653 — item 1, Q1) #3724 structural pin drops its exclusion (eightpeak*, avg*pair calls per detector body, no peak-only call survives). New ungated pinWaitRateWindow_ReadsThePeakAndMeanPair_AndTheRobustArmIsTheSharedPairGate_LiteVerbatim: the four columns in the reader's ordinal order on the PG const and in Lite's text; the gate call namesHeavyTailModifiedZThresholdandWaitProfileFallbackMsPerSec; the inlinemodifiedZ < HeavyTailModifiedZThresholdgate is gone from both SKUs; the ratio arm testsmeanRatio; the no-baseline arm still testspeakRatealone. New liveEndToEnd_WaitProfileDetector_PeakAndMeanPair_OneHotCollectionStaysQuiet_SustainedWindowFires_AgainstDevPostgres: three Mondays of rates cycling 100/200/300 ms/sec → Full bucket trustworthy, median 200, MAD 100 (percentile_cont hand-checked in the doc comment), robust σ 148.26; window A (fifteen at the median, one at 3200 ms/sec — peak 20.2σ over the floor, mean 1.26σ) → NO fact; window B (fifteen at 1500, one at 3200 — mean 9.5σ) → oneANOMALY_WAIT_PROFILEwithcurrent_ms_per_sec3200,avg_ms_per_sec1606.25, both modified z stamped, ratio the peak's. The pre-existing is_new e2e gainsavg_ms_per_secassertions. Red-first shown: with the PG gate stubbed back to peak-only, window A fires and the structural pin fails (2 red); restored, 18/18.Darling.Tests/RobustBaselineTests— 23/23; newWaitProfilePair_TheStoryNamesTheMeanBesideThePeak_AndTheScorerStillGradesThePeaks(gate-arm prose with the robust σ, ratio-arm prose with the multiple, is_new prose without the "fires only when" claim, legacy sentence verbatim, scorer grade identical before/after the pair).Lite.Tests/AnomalyDetectorTests— 28/28 through the real DuckDB detector; 3 new: one hot collection in a quiet window stays quiet (with the red-first arithmetic asserted in-test); sustained window fires with peak/mean/both z stamped; no-baseline fallback still fires on the peak alone with a mean under the bar.SeedWaitStatAsyncgains an optional storedsampleIntervalSeconds(null default, every existing caller unchanged) so the fixtures rate deterministically on the v60 column. Red-first shown: with the Lite gate stubbed back to peak-only, exactly the hot-collection fixture fails; restored, 28/28.RepoFileAdoptionTests2/2,StragglerCommandTimeoutTests48/48,LiveCleanupConversionRatchetTests15/15,MeasurementContractCensusTests7/7 (the newAVG(CASE WHEN interval_sec > 0 …)arm carries the guarded opener the census demands),McpPageContractTests43/43,FleetIdentifierScrubTests2/2,DocCommentHygieneTests76/77 (the one red is the documented harness artifact — the throwaway csproj in the tree). LiveDarlingAnalysisPipelineTests11/11 andAnalysisAsOfAnchorTests3/3 (the is_new arm through the full pipeline).-c Release -p:EnableWindowsTargeting=true, 0 warnings, 0 errors.CI-first: nothing — every touched path executed here.
Closes #3741