Repository navigation
Extreme, corroborated anomaly findings can now cross the notify floor, so the baseline engine is no longer notification-inert at shipped settings (Fixes #3526) - #3584
Conversation
) Every ANOMALY_* ramp saturates its base at 1.0, the Layer-3 tuning-class cap holds the final at 1.49, no amplifier arm existed, and the shipped notify floor is 1.5 — so the baseline engine could never page at default settings. FactScorer: - IsExtremeAnomaly: a per-fact escape from the tuning-class cap for ANOMALY_* only, at 3x the cutoff the fact fired at (10.5σ robust, 15σ heavy-tail, 6σ classical; 3x the absolute bar on the low-quality fallback path, never for is_new wait profiles; ratio/count/delta families stay capped). - AnomalyAmplifiers: a co-fire arm — sibling anomaly families and measured absolute facts corroborate at +0.2/+0.3, so an extreme anomaly with two corroborators scores 1.6 and a lone or routine one cannot reach 1.5. - The impact-peer escape and CXPACKET's cap are unchanged. Lite.Tests/FactScorerTests: 12 new pins on the escape bar arithmetic per path, the co-fire arm, the routine/lone/single-corroborator negatives, the low-quality sigma blindness, CXPACKET staying capped beside extreme anomalies.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| anomaly cannot escape on an untrustworthy baseline at all, which is correct: "we do not know your | ||
| normal yet" is not evidence of an outage. An operator who scales a metric's deviation threshold | ||
| scales its fire_threshold with it (ModifiedZThresholdFor), so the escape bar tracks the knob. */ | ||
| private const double ExtremeAnomalyMultiple = 3.0; |
There was a problem hiding this comment.
Nit (non-blocking): the extremity escape's reach is bounded by SigmaDisplayCap (25σ), but the anchor it's multiplied against (FireAnchor) tracks the operator's SetDeviationThreshold/ModifiedZThresholdFor knob unbounded. If an operator raises a metric's deviation threshold past roughly 8.3x the default (e.g. robust cutoff scaled above ~8.3, so 3 * anchor > 25), deviation_sigma can never reach 3 * FireAnchor(fact) because it's clamped at 25 before this check ever sees it — the escape becomes permanently unreachable for that metric, silently reproducing the exact "structurally quietest" bug this PR fixes, just for a differently-tuned deployment instead of the shipped default. Worth a guard (e.g. cap the escape multiplier's effective threshold at SigmaDisplayCap) or at least a doc note next to ExtremeAnomalyMultiple calling out the interaction with SigmaDisplayCap for operators who scale thresholds up significantly.
There was a problem hiding this comment.
Good catch — fixed in 20d0076. AnomalyGate clamps the stored deviation_sigma at SigmaDisplayCap (25σ) on both detectors before the scorer ever sees it, so an anchor past ~8.3 made 3 × anchor unreachable and the escape silently dead for that metric — the same defect this PR exists to fix, just for a hard-tuned store. IsExtremeAnomaly now takes min(3 × anchor, SigmaDisplayCap) on the deviation-scored path (a sigma pinned at the cap means "at least 25σ", extreme under any anchor an operator can set); the ExtremeAnomalyMultiple comment documents the interaction. The wait profile's modified_z is not display-capped (BaselineMath.ModifiedZScore) so it needs no bound. Pinned with two new Theory rows: anchor 10.0 at 25.0σ → released (1.6), at 24.99σ → capped (1.49); both values executed against the built library.
There was a problem hiding this comment.
LGTM — reviewed FactScorer.cs's new anomaly-extremity escape and co-fire amplifier arm against the shared AnomalyGate metadata contract (Lite AnomalyDetector + Darling PgAnomalyDetector both emit identical fields), hand-verified the new/changed FactScorerTests numbers, and confirmed no SQL or Lite/Darling parity drift (single shared project, no per-app fork). Left one non-blocking inline note about the escape's interaction with SigmaDisplayCap under heavily-scaled operator thresholds.
AnomalyGate clamps the stored deviation_sigma at 25σ before the scorer sees it, so an operator-scaled anchor above ~8.3 made 3x unreachable and the escape silently dead for that metric — the defect this PR fixes, for a tuned store. IsExtremeAnomaly now takes min(3x anchor, SigmaDisplayCap); two Theory rows pin anchor 10.0 at 25.0 (released) and 24.99 (capped).
|
Reviewed. Summary of what I checked:
No correctness, parity, or security issues found. |
There was a problem hiding this comment.
LGTM — verified the extremity-escape and co-fire-amplifier arithmetic against the detectors' shipped metadata/thresholds, confirmed the change is confined to the single shared FactScorer used by both Lite and Darling (no parity risk), and found no correctness or security issues.
…CTEs over the pg_* raw hypertables under the one RobustTierScaffold (TPS and deadlocks/hour off the reset-aware per-database difference, the capture's total_sessions, the Aurora all-types wait rate under the three-state interval with CPU excluded, percent of the configured ACU ceiling), five detectors through the shared AnomalyGate writing the same metadata the #3584 ramp and extremity escape read, a PostgreSQL ratio ramp with no sentinel for a first occurrence, the load-family co-fires confirmed by a capacity reading and never a raw one, PG_CPU_PERCENT graded only when the capacity percent was measured, and advice in PostgreSQL nouns (#3542 lane 9) (#3689) Unmeasured floors and bars carry their lineage and every anomaly fact carries threshold_lineage = 0. Stock PostgreSQL's sampled wait estimate gets no anomaly baseline in v1.
1 participant
The baseline engine's alarm could not ring at shipped settings
The robust-statistics half of the analysis engine — hour-of-week median/MAD baselines, trustworthiness tiers, the #1743 modified-z gate — produces
ANOMALY_*findings whose final severity could not reach the notification floor. Three facts aboutFactScorercombined into a structural ceiling:ANOMALY_*ramp saturates its base at 1.0 (ScoreAnomalyFact: 0.5 at the fire threshold, 1.0 at 2x it).GetAmplifiershad noANOMALY_*arm, so final = base ≤ 1.0 — no corroboration could ever lift one.IsTuningClassKeyputs everyANOMALY_*under the Layer-3TuningClassSeverityCeiling = 1.49, released only when THREADPOOL fired, SOS ≥ 25 % of period, or RESOURCE_SEMAPHORE ≥ 10 % co-fired.The shipped
notify_severityis 1.5 in both editions. So a 20σ session spike beside a 15σ batch-request spike and 85 % CPU at 3am produced nothing that could page — the product's best per-server-calibrated statistics were structurally its quietest. The 3.8 changelog's own #3467 narrative shows the shape in the wild: the timely anomaly findings on that pileup were correct and capped under the gate (ANOMALY_BATCH_REQUESTS1.0 against 1.5).What the product does now: an anomaly whose deviation is extreme against the cutoff it fired at is released from the tuning-class cap on its own evidence, and a new anomaly co-fire amplifier arm lets corroboration carry it into the CRITICAL band. Extreme + two corroborators scores 1.6 and reaches the operator. Routine, lone, or singly-corroborated anomalies cannot reach 1.5. CXPACKET and the impact-peer escape are untouched.
Why this shape (and what was rejected)
The issue offered three shapes: extend the escape so a confirmed-severe anomaly can cross 1.5; an anomaly-specific notify tier; or document-and-default
notify_severityinto the anomaly band. This PR is the first — the campaign engine review's recommendation — for reasons that are the cap's own reasoning read forward:ANOMALY_*breadth was wrong. The Recs Stage B (severity): cap tuning-class CXPACKET, escalate meltdowns through THREADPOOL #1494 cap exists because CXPACKET stacks amplifiers into CRITICAL on parallelism alone, and its impact-peer escape (thread/CPU/grant starvation) is the correct test for that key. Applied wholesale toANOMALY_*it asked the wrong question: an anomaly is not a tuning opportunity that needs a peer to certify impact — it is a per-server statistical deviation whose own magnitude is the evidence. So the second escape is per-fact and reads the anomaly's own statistic; the first escape is unchanged and CXPACKET / CXCONSUMER / QUERY_HIGH_DOP have no extremity arm (pinned).DarlingConfig.NotifySeverity,PgMigrations, theDarlingWorkernotify path and Lite'sAnalysisNotifySeverityare not touched.The extremity escape —
IsExtremeAnomaly, bar = 3x the fire thresholdEvery deviation ramp saturates at 2x its anchor, so 3x sits a full anchor past the point where the scorer ran out of scale — "extreme" means "the ramp stopped distinguishing a while ago", not "the top of the ramp". Per path, with the shipped
AnomalyThresholdscutoffs:baseline_low_quality = 1)All sigma bars sit under the 25σ
SigmaDisplayCap, so a fact can actually carry them. The 6σ classical bar is defensible against the #1743 fleet measurement, which read a busy tenant's real 2-3x evening surge at 1.4-2.0 classical sigmas: six of them against a stddev history has already inflated is a rare reading. The escape reads the same metadata the ramp grades from —deviation_sigmavsfire_threshold,fallback_exceedanceon the fallback path,modified_zthenratiofor the wait profile — via a sharedIsDeviationScoredAnomalyKey/FireAnchor, so a fact can never be extreme on one statistic while scored on another. Both detectors (PgAnomalyDetector, LiteAnomalyDetector) emit identical metadata names off the sharedAnomalyGate.ZDecision, confirmed by grep, so the shared scorer routes both SKUs identically.Deliberate exclusions, each pinned:
is_newwait profiles never escape — theNoBaselineRatiosentinel (100) is a scoring device, not a measurement.ANOMALY_WAIT_) — none is graded in sigmas, so "3x the fire threshold" has no calibrated meaning, and the blocking/deadlock classes already reach CRITICAL through their never-capped impact keys.The co-fire arm —
AnomalyAmplifiersModelled on the impact-peer style and the PAGEIOLATCH / IO-latency arms. A sibling anomaly family firing in the same window (
BaseSeverity > 0— the scorer zeroes anything under its cutoff) is a co-fire; a measured absolute fact confirming the same pressure is a co-fire, at the bars the existing arms already use (SQL CPU ≥ 80 % as in the SOS / compile-gateway arms;IO_*_LATENCY_MSat its concerning bar; PAGEIOLATCH 10 %, WRITELOG 5 %,MEMORY_GRANT_PENDING ≥ 1as in their own arms). Families: load (sessions / batch / CPU / query duration corroborate one another, +0.3 each, self-key omitted); read and write latency (the absolute latency fact, the wait profile, the other side, the matching wait); wait profile (I/O latency, query duration +0.3; load +0.2); memory pressure (ring-buffer notifications, grant waiters, RESOURCE_SEMAPHORE, PAGEIOLATCH, read latency).Worked numbers (in the code comment too):
Blast radius
PerformanceMonitor.Analysis/FactScorer.csis shared by Lite, Darling and the deprecated Dashboard, so all three change identically — that is the one-truth property, not a risk. Base severities are byte-identical for every anomaly (thefire_thresholdanchor logic moved intoFireAnchorunchanged); every existingFactScorerTestspin inLite.Testsanddeprecated/Dashboard.Testsand everyRobustBaselineTestsscorer pin holds (verified by executing the scenarios, below). Story severity is the root fact's severity (InferenceEngine), so an anomaly root that now scores 1.6 rides the existing notify / mute / occurrence-folding / cooldown machinery with no new delivery semantics.AmplifierResultsgain entries for anomaly facts; the MCP analysis surfaces render that list generically.What this does NOT do
notify_severitydefaults, the notify path, or any config surface.AnomalyGate, orAnomalyThresholds— no threshold moves; the escape bar tracks an operator'sSetDeviationThresholdknob automatically becausefire_thresholdis already knob-scaled.deprecated/Dashboard.Tests— its shared-scorer pins are unchanged and still hold.Test plan
New pins —
Lite.Tests/FactScorerTests.cs, section "Layer 3: #3526 anomaly extremity escape + the anomaly co-fire amplifier arm" (12 tests, 22 cases):Score_ExtremeCorroboratedAnomaly_CrossesNotifyFloor— the issue's shape: 20σ sessions + 15σ batch + CPU 85 % → 1.6 both sides, with the matched amplifier descriptions asserted.Score_AnomalyAtFireThreshold_WithCoFires_StaysWarning— same set, root at exactly 3.5σ → 0.8.Score_SaturatedButRoutineAnomaly_WithThreeCoFires_CappedAtWarningCeiling— 7σ + three co-fires → 1.9 pre-cap → 1.49.Score_AnomalyExtremityEscape_OpensAtThreeTimesTheFireThreshold(Theory ×6) — 6.0 / 5.99 classical, 10.5 / 10.49 robust, 15.0 / 14.99 heavy-tail → 1.6 / 1.49 either side of each bar.Score_LowQualityAnomaly_EscapesOnlyAtThreeTimesItsAbsoluteBar_NeverOnSigma(Theory ×3) — planted 30σ ignored; exceedance 2.5 / 2.99 → 1.49, 3.0 → 1.6.Score_CxPacketWithExtremeAnomalyCoFires_StillCapped_AnomaliesReleased— CXPACKET 1.49, the anomalies 1.6, in one fact set.Score_LoneExtremeAnomaly_StaysAtBase_DoesNotPage— 25σ alone → 1.0, no amplifier matched.Score_ExtremeAnomaly_SingleCoFire_StaysWarning→ 1.3.Score_AnomalyCoFireArm_DoesNotSelfCorroborate;Score_WaitProfileAnomaly_EscapesAtThreeTimesTheHeavyTailCutoff(15.0 → 1.6, 14.99 → 1.49);Score_IsNewWaitProfile_NeverEscapes_DespiteSentinelRatio→ 1.49;Score_BlockingSpikeAnomaly_NoEscape_NoArm→ 1.0, empty arm.No existing test was modified or deleted: no current pin asserted an anomaly's final severity ceiling, so nothing needed a deliberate update (searched both test projects for
TuningClassSeverityCeiling,IsTuningClassKey,GetAmplifiers,ScoreAnomalyFact,1.49,ANOMALY_).Verification on this machine (macOS; the test projects are
net10.0-windowsand cannot execute here):dotnet build PerformanceMonitor.Analysis/PerformanceMonitor.Analysis.csproj -c Debug -v q -nologo— 0 warnings, 0 errors.dotnet build Lite.Tests/Lite.Tests.csproj -c Debug -v q -nologo -p:EnableWindowsTargeting=true— 0 warnings, 0 errors.dotnet build Darling/Darling.Tests/Darling.Tests.csproj -c Debug -v q -nologo -p:EnableWindowsTargeting=true— 0 warnings, 0 errors.net10.0, every scenario the new tests pin — all 22 cases plus the pre-existing CXPACKET-alone 1.49, low-quality floor 0.5, anchored-ramp 0.5 and legacy 0.75 pins — was executed against the built DLL from a throwaway console project outside the worktree; every value matched the pinned expectation exactly (the 14.99σ wait-profile row lands at base 0.9995 → 1.5992 pre-cap → 1.49, as designed). The xunit tests themselves first execute in CI.Which component(s) does this affect?
How was this tested?
See the test plan: compile-verified in all three projects with zero warnings; scorer arithmetic executed against the built shared library for every pinned case; xunit execution is CI's.
Checklist
dotnet build -c Debug)Fixes #3526