Repository navigation
Anomaly engine Stage A: baseline-quality gate, wait-profile redesign, honest no-baseline rendering - #1491
Merged
Conversation
… + honest no-baseline rendering Builds on the SAFE #1486 floors/sigma-cap. Three coordinated changes across the triplicated anomaly engine (Lite AnomalyDetector/BaselineProvider, Darling PgAnomalyDetector/PgBaselineProvider, Dashboard SqlServer*) plus shared PerformanceMonitor.Analysis (FactScorer, FactAdvice, new AnomalyGate): Change 2 - baseline quality gate: BaselineBucket gains DistinctDays (COUNT(DISTINCT date), summed on hour-only collapse, MAX-proxied on flat) and IsTrustworthy (tier-aware distinct-day mins Full>=3/HourOnly>=10/Flat>=15 + real dispersion). Bounded metrics (CPU/mem/IO) gain an absolute dispersion floor (AbsStdDevFloor) folded into EffectiveStdDev. Shared AnomalyGate.EvaluateZScore centralizes the interaction trap: trustworthy -> trust z with the #1486 magnitude floor as a sanity ceiling; thin baseline -> fall back to a HIGHER absolute bar (CPU>=90, mem>=95, batch>=5000, sessions>=500, query-dur>=5s, IO>=50ms), never silence. Change 1 - wait redesign: per-type ANOMALY_WAIT_<type> facts (per-hour-vs- per-interval unit inflation; missed minority waits) become ONE ANOMALY_WAIT_PROFILE - all-types ms/sec (interval via LAG) vs the WaitMsPerSec baseline, top types as contrib_<TYPE> metadata. Dashboard SqlServerMetricNames gains WaitMsPerSec + its T-SQL baseline query. FactScorer gains an ANOMALY_WAIT_PROFILE arm; DefaultRatioThreshold + slope retuned to the honest per-second scale (4x->0.5, 12x->1.0). FactAdvice composes the profile. 100x honest-rendering fix: the blocking/deadlock no-baseline sentinel now sets an is_new flag; the composer renders "first occurrence / no baseline yet" instead of a fake "spiked to 100x". Tunable ratio/slope/fallback consts labelled to calibrate on SQL2025/HammerDB. New tests across all three suites; the 100x/per-type ScenarioTests + Darling live-PG arm updated to the profile. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Finding 1 (HIGH): the baseline quality-gate fallback fired but FactScorer silently zeroed it. The 7 bounded z-metric arm gated severity on deviation_sigma (if deviation < 2.0 return 0.0), but on a thin-baseline fallback fire that stored sigma is the real (small) z, so a genuine problem (e.g. memory 96% on a young store) fired the gate yet produced NO finding (InferenceEngine drops severity-0 facts). AnomalyGate.ZDecision now carries FallbackExceedance (peak / the absolute-fallback bar); FactScorer, when baseline_low_quality=1, grades off that exceedance (floor 0.5 to clear InferenceEngine's 0.5 entry-point, ramping to 1.0 at 2x the bar) instead of applying the 2-sigma gate. The trustworthy z-path is byte-for-byte unchanged. Detectors plumb fallback_exceedance at all 7 emit sites across the 3 providers. Finding 2 (HIGH): DarlingAnalysisPipelineTests planted a 2 x 100000ms window (~166.7 ms/sec), below the 250 ms/sec fallback bar, so nothing fired and Assert.Single threw after change 1 replaced per-type wait facts with one ANOMALY_WAIT_PROFILE. Reworked to mirror its already-fixed sibling DarlingAnomalyBaselineTests: 3 x 200000ms at 5-minute spacing (~666.7 ms/sec) and assert ANOMALY_WAIT_PROFILE. Live-PG gated (skips locally); fixed by construction. Finding 4 (MEDIUM): FlatDayMin=15 was structurally unreachable (CollapseToFlat sets DistinctDays = MAX of per-hour-bucket distinct days, capped near ~5 in a 30-day window), leaving the Flat trust branch permanently dead. Lowered to 3 (byte-identical across all 3 BaselineBucket classes) and corrected the now-wrong "conservative lower bound" comments (the MAX is a ~5 ceiling). Findings 3 (blocking/deadlock count-based firing) and 5 (is_new sentinel tuning) intentionally untouched per instructions (owner design decisions). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stage A of the recs-remainder redesign (analysis/anomaly engine)
Builds on the SAFE subset already merged in #1486 (absolute-magnitude floors + sigma display cap) — this is the remainder's Stage A. Scope is strictly Stage A: Change 2 (quality gate) then Change 1 (wait redesign), plus the 100x honest-rendering fix on the blocking/deadlock detectors. Stage B (incident dedup + CXPACKET severity cap) is deliberately NOT here.
3-provider parity change across the triplicated engine + shared
PerformanceMonitor.Analysis.Change 2 — baseline quality gate + bounded dispersion floors
BaselineBucketgainsDistinctDays(COUNT(DISTINCT collection_time::DATE)per metric — SUM through the hour-only collapse (exact), MAX proxy through the flat collapse) andIsTrustworthy(tier-aware distinct-day minimums Full >=3 / HourOnly >=10 / Flat >=15, plus real dispersion).AbsStdDevFloor, folded intoEffectiveStdDev) so a variance-collapsed baseline can't manufacture a giant z.AnomalyGate.EvaluateZScorecentralizes the interaction trap so the three detectors can't drift: trustworthy → trust z with the Anomaly detector: magnitude floors + sigma display cap to kill young-store noise #1486 magnitude floor as a sanity ceiling; thin/young → fall back to a higher absolute bar (CPU >=90%, mem >=95%, batch >=5000/s, sessions >=500, query-dur >=5s, IO >=50ms), never silence.Change 1 — wait redesign → one
ANOMALY_WAIT_PROFILEANOMALY_WAIT_<type>facts (per-hour-vs-per-interval unit inflation; missed a minority-but-real wait) become ONEANOMALY_WAIT_PROFILE: whole-server all-types wait ms/sec (interval viaLAG, never an assumed cadence) vs theWaitMsPerSecbaseline, with the top wait types ascontrib_<TYPE>metadata.SqlServerMetricNamesgainsWaitMsPerSec+ its T-SQL baseline query.FactScorergains anANOMALY_WAIT_PROFILEarm;DefaultRatioThreshold+ slope retuned to the honest per-second scale (4x → 0.5, 12x → 1.0).FactAdvicecomposes the profile from its contributors.100x honest-rendering fix (blocking/deadlock)
is_newflag; the composer renders a first occurrence ("no baseline yet") instead of a fake "spiked to 100x".All ratio/slope/fallback constants are labelled tunable — calibrate on SQL2025/HammerDB.
Testing
ThemeParityTests(theme XAML palette drift between Dashboard/Lite; confirmed failing onorigin/dev, and this diff touches zero XAML).Updated tests (intentional, per spec): the two
ScenarioTestswait-spike assertions (ANOMALY_WAIT_PAGEIOLATCH_SH→ANOMALY_WAIT_PROFILE+ contributor) and the Darling live-PG detector arm (now theis_newprofile, since a single-day history is untrustworthy under the gate). New tests added: thin-baseline fallback-fires-but-suppresses-sub-bar-z, healthy-baseline-trusts-z, DistinctDays population + collapse, wait-profile-one-fact + minority capture, and the profile scoring ramp.Do not merge — Stage B is a separate change.
🤖 Generated with Claude Code