Repository navigation
issue-3653 A8 option B: tier selection falls back to a coarser trustworthy tier - #4180
Conversation
…y tier A newly monitored target's Full (hour+dow) bucket clears its sample-count floor on day one but not its 3-distinct-day floor for about 3 weeks, so every z-score detector was dark until then even when the coarser HourOnly or Flat tier was already trustworthy. SelectBucket now computes today's tier-by-sample-count choice unchanged, returns it immediately when it is trustworthy or zero-history, and otherwise walks only the coarser tiers (built exactly as today, sentinel or pooled) for the first one that is IsTrustworthy. If none is, returns the original chosen bucket unchanged (today's absolute-fallback path). Adds 6 RobustBaselineTests covering: exact-sentinel HourOnly fallback, fallback through to Flat, all-untrustworthy (absolute fallback preserved), IsZeroHistory never overridden, trustworthy Full unchanged (same instance), and pooled-synthesis fallback with no sentinels.
DocCommentHygieneTests caught two stacked <summary> openings on the same member. Merged the #3653 addition into the existing summary as another <para> block, same content.
BaselineBucket.IsYoung is true only for the untrustworthy-for-too-few- distinct-days case the tier walk exists for: real dispersion and enough samples, but too few DISTINCT days for this tier. Never true for a trustworthy bucket, a zero-dispersion bucket (constant-valued fixtures, idle metrics), or a thin (too-few-sample) bucket. BaselineMath.SelectBucket now walks coarser only when !chosen.IsYoung fails to hold, i.e. only for young buckets, instead of walking on !IsTrustworthy && !IsZeroHistory (which also fired on zero-dispersion and thin buckets and broke 3 Lite tier-selection tests in CI). Adds two RobustBaselineTests: a zero-dispersion Full bucket with a trustworthy coarser tier present (not walked, returns Full), and a boundary case pinning that the walk fires on distinct-days alone, not on a sample-count deficit (a genuinely thin Full bucket is never selected as Full in the first place, so IsYoung's sample floor can only be observed at the floor's boundary).
…stinct days Each failing fixture's chosen bucket was IsYoung under the narrowed tier walk (real dispersion, enough samples, too few distinct days for its tier), so the walk correctly moved it coarser to Flat — a by-design change, not a bug. Rather than flip the expected tier, give each fixture enough distinct days to keep pinning what it was written for: - BaselineProviderTests.GetBaseline_SparseBucket_CollapsesToHourOnly: 9 further distinct days added so the pooled hour-only tier clears its own 10-day floor and is not itself walked past to Flat. - BaselineProviderTests.GetBaseline_HourOnly_SumsDistinctDaysAcrossDowBuckets: 4 distinct days -> 10, for the same reason (still pins the SUM behavior across dow buckets, at the day count HourOnly's own floor requires). - LocalClockBucketKeyTests.Provider_ZoneId_EstAndEdtRowsAcrossTheSpringForward_ShareOneLocalBucket: a third distinct Tuesday added to each of the two DST-straddle buckets so DistinctDays clears Full's 3-day floor; still pins the EST/EDT keying and the rollover dating, unchanged.
|
Lane L1e-2 done. Narrowed the tier walk to YOUNG buckets only, per the ruling. Changes (91a978e, 9403808 on feat/3653-b-trusted-tier):
Verified: Darling build ( Not verified: full Darling suite with |
Coordinator's diagnosis: this test pins DST KEYING (which bucket the rows land in), not tier selection, and the young-target walk (this PR) correctly moves both the CPU and the rollover Full bucket to Flat -- CPU's bucket has real spread across two Tuesdays (DistinctDays 2 < Full's day floor 3); the rollover bucket clears IsYoung too, not from real spread (it's ten identical 99s) but because CPU's 5.0 absolute dispersion floor keeps EffectiveStdDev above zero regardless. Both walks fall through HourOnly (equally thin) to the same Flat bucket, which pools all 22 samples across three distinct Tuesdays and clears its 3-day floor. Read the RAW Full bucket through GetBucketMapAsync's dictionary (map.Buckets[(hour, dow)]) for the keying assertions the test actually cares about, and added separate assertions pinning the new walk behaviour (IsYoung on both raw buckets; Tier == Flat, IsTrustworthy, and the pooled SampleCount on the walked GetBaselineAsync result).
Part of #3653
What changed
BaselineMath.SelectBucketpicked a tier (Full → HourOnly → Flat) purely bysample count, then handed the chosen bucket back even when
BaselineBucket.IsTrustworthywas false for it. At a 5-minute collectioncadence, the Full (hour+dow) tier clears its sample floor (10-14 samples) on
day one, but Full's day floor is 3 DISTINCT same-weekday dates — one per
calendar week — so it stayed selected-but-untrustworthy for about three
weeks on a newly monitored target, even though the coarser HourOnly tier
(day floor 10, ~10 days) or Flat tier (day floor 3, ~3 days) was already
trustworthy. Every z-score detector was routed to the much higher
absolute-threshold bar for that whole window.
SelectBucketnow:chosen).chosenunchanged if itIsTrustworthyORIsZeroHistory— maturetargets are byte-identical, and the zero-history arm (A month of zeros is the strongest baseline there is, not the absence of one: a zero-history bucket now fires as an extremity instead of routing to the young-store fallback bar (#3691) #3849, measured by
PostgreSQL-target analysis engine: v2/v3 slices and the v1 lanes' residue (from #3542) #3691 D8) is never overridden.
chosen(HourOnly then Flat),building each exactly as today (the provider's exact GROUPING SETS
sentinel when present, else pooled synthesis, gated by the same
CollapseThreshold/3 sample floors), and returns the first one that IStrustworthy.
chosen, unchanged — today'sabsolute-fallback path and low-quality metadata.
All three consumers (
PgBaselineProvider.LookUp, LiteBaselineProvider,BaselineBucketMap.For) go throughSelectBucket, so this one changecovers every surface. Thresholds, tier floors,
IsTrustworthyand everycaller are unchanged.
Touched only
BaselineMath.csandRobustBaselineTests.cs, per brief.Tests added (
Darling/Darling.Tests/RobustBaselineTests.cs)SelectBucket_ChosenFullUntrustworthy_ExactHourOnlyTrustworthy_ReturnsHourOnlySelectBucket_FullAndHourOnlyUntrustworthy_ExactFlatTrustworthy_ReturnsFlatSelectBucket_EveryTierUntrustworthy_ReturnsChosenFull_TheAbsoluteFallbackAsTodaySelectBucket_ChosenFullIsZeroHistory_NeverReplacedByACoarserTierSelectBucket_TrustworthyFull_ReturnedUnchanged_MatureTargetIsByteIdenticalSelectBucket_NoSentinels_PooledFullUntrustworthy_PooledHourOnlyTrustworthy_ReturnsPooledHourOnlyAll pre-existing
SelectBuckettests are unchanged and still pass — none oftheir fixtures fell in the 3–20-day young-target band this lane targets.
A doc-comment collision:
DocCommentHygieneTests.NoMemberCarriesTwoStackedSummaryBlockscaught two stacked
<summary>blocks I'd left onSelectBucket(the newrationale as a second
<summary>instead of another<para>in theexisting one). Fixed by merging it in as a
<para>, verified green in afollow-up commit (4195e82).
What I ran
dotnet buildonPerformanceMonitor.Analysis,Darling/Darling.Tests,Lite/PerformanceMonitorLite.csproj,Lite.Tests/Lite.Tests.csprojwith-p:EnableWindowsTargeting=true— all green, 0 warnings, 0 errors, afterthe doc-comment fix.
timescale/timescaledb:2.28.1-pg18,DARLING_TEST_PGset):RobustBaselineTests(35),LocalClockBucketKeyTests(13),
AnomalyGateNullWindowMonteCarloTests(11),AnomalyGateTilesTests(17),
PgTargetAnomalyTests(44),DarlingAnomalyBaselineTests(18),DarlingAnalysisStoreTests(9),DarlingAnalysisBudgetTests(3),DarlingAnalysisServiceEngineRoutingTests(11),DarlingAnalysisPipelineTests(11) — all green, 0 failed, 0 skipped.AnomalyNullWindowLiveTests(still[Fact(Explicit = true)],HistoryDays = 10), run with-explicit on -showLiveOutputon bothorigin/devand thisbranch against the same live rig:
dev:DIAG shiftFires4h=0 shiftPasses4h=3 shiftFires24h=0 shiftPasses24h=3 spikeAlone4h=0 spikeAlone24h=0 null4h=0/57 null24h=0/12Honest read: no change at 10 days of history for this particular fixture's
pattern — its z-path outcome here isn't gated by tier trust in the way that
moved. I'm reporting the DIAG lines as instructed rather than reading a
result into them.
DARLING_TEST_PGunset:Total: 13671, Errors: 0, Failed: 214, Skipped: 730, Not Run: 2. Diffedthe failing-test names byte for byte against the same run on
origin/dev(
Total: 13665, Failed: 214) — identical failing-test set, allpre-existing WPF/Windows-only tests that don't run on this Mac. This
branch adds 6 new tests (the ones above) with zero new failures and zero
regressions.
Lite/PerformanceMonitorLite.csprojandLite.Tests/Lite.Tests.csprojbuilt clean with
-p:EnableWindowsTargeting=true. Did not runLite.Tests(hangs on this Mac per brief); left to CI.Watch in CI (Lite.Tests, 3–20 day seeded-history fixtures asserting the absolute path)
AnomalyDetectorTests.DetectCpuAnomalies_ThinBaseline_AbsoluteFallbackFires_NotSilentand
AnomalyDetectorTests.DetectCpuAnomalies_ThinBaseline_SuppressesPureZSpikeBelowAbsoluteBar— both use
SeedThinBaselineCpu(2 distinct days, 16 samples). By thislane's rule this Full bucket is untrustworthy, and there's no HourOnly or
Flat sentinel/pooled data seeded in that fixture, so both stay on the
absolute-fallback path exactly as asserted. Flagging anyway since they're
exactly the shape this lane changes behavior for.
AnomalyDetectorTestsalso hasSeedThinBaselineWaitRates(2 distinctdays) feeding wait-rate anomaly tests — same reasoning, same expected
outcome (no seeded coarser tier to fall back to, so unchanged).
ComparisonBandingTests.AnUntrustworthyBaseline_FallsBackToTheAbsoluteRuleconstructs a single untrustworthy
BaselineBucketdirectly (not throughSelectBucket/a map) and asserts onComparisonBanding, not on tierselection — unaffected by this change, but flagging since it lives in the
same 1-distinct-day territory.
coarser sentinel/pooled tier IS present and trustworthy, so I did not
expect (and did not find, by build) any Lite.Tests compile-time
expectation change.
Not verified
level
SelectBucketfixtures and the oneAnomalyNullWindowLiveTestsdemonstration, which showed no delta at 10 days for its specific fixture).
CHANGELOG entry
On a newly monitored server, anomaly detectors use the hour-of-day or overall baseline while the hour-of-week one is still too young to trust, instead of only the absolute threshold. Anomaly detection now starts in days rather than about three weeks (#3653).