Skip to content

issue-3653 A8 option B: tier selection falls back to a coarser trustworthy tier - #4180

Merged
erikdarlingdata merged 5 commits into
devfrom
feat/3653-b-trusted-tier
Sep 24, 2026
Merged

erikdarlingdata merged 5 commits into
devfrom
feat/3653-b-trusted-tier

Conversation

@erikdarlingdata

Copy link
Copy Markdown
Owner

Part of #3653

What changed

BaselineMath.SelectBucket picked a tier (Full → HourOnly → Flat) purely by
sample count, then handed the chosen bucket back even when
BaselineBucket.IsTrustworthy was false for it. At a 5-minute collection
cadence, the Full (hour+dow) tier clears its sample floor (10-14 samples) on
day one, but Full's day floor is 3 DISTINCT same-weekday dates — one per
calendar week — so it stayed selected-but-untrustworthy for about three
weeks on a newly monitored target, even though the coarser HourOnly tier
(day floor 10, ~10 days) or Flat tier (day floor 3, ~3 days) was already
trustworthy. Every z-score detector was routed to the much higher
absolute-threshold bar for that whole window.

SelectBucket now:

  1. Computes today's choice by sample count exactly as before (chosen).
  2. Returns chosen unchanged if it IsTrustworthy OR IsZeroHistory — mature
    targets are byte-identical, and the zero-history arm (A month of zeros is the strongest baseline there is, not the absence of one: a zero-history bucket now fires as an extremity instead of routing to the young-store fallback bar (#3691) #3849, measured by
    PostgreSQL-target analysis engine: v2/v3 slices and the v1 lanes' residue (from #3542) #3691 D8) is never overridden.
  3. Otherwise walks only the tiers COARSER than chosen (HourOnly then Flat),
    building each exactly as today (the provider's exact GROUPING SETS
    sentinel when present, else pooled synthesis, gated by the same
    CollapseThreshold/3 sample floors), and returns the first one that IS
    trustworthy.
  4. If none is, returns the original chosen, unchanged — today's
    absolute-fallback path and low-quality metadata.

All three consumers (PgBaselineProvider.LookUp, Lite BaselineProvider,
BaselineBucketMap.For) go through SelectBucket, so this one change
covers every surface. Thresholds, tier floors, IsTrustworthy and every
caller are unchanged.

Touched only BaselineMath.cs and RobustBaselineTests.cs, per brief.

Tests added (Darling/Darling.Tests/RobustBaselineTests.cs)

  • SelectBucket_ChosenFullUntrustworthy_ExactHourOnlyTrustworthy_ReturnsHourOnly
  • SelectBucket_FullAndHourOnlyUntrustworthy_ExactFlatTrustworthy_ReturnsFlat
  • SelectBucket_EveryTierUntrustworthy_ReturnsChosenFull_TheAbsoluteFallbackAsToday
  • SelectBucket_ChosenFullIsZeroHistory_NeverReplacedByACoarserTier
  • SelectBucket_TrustworthyFull_ReturnedUnchanged_MatureTargetIsByteIdentical
  • SelectBucket_NoSentinels_PooledFullUntrustworthy_PooledHourOnlyTrustworthy_ReturnsPooledHourOnly

All pre-existing SelectBucket tests are unchanged and still pass — none of
their fixtures fell in the 3–20-day young-target band this lane targets.

A doc-comment collision: DocCommentHygieneTests.NoMemberCarriesTwoStackedSummaryBlocks
caught two stacked <summary> blocks I'd left on SelectBucket (the new
rationale as a second <summary> instead of another <para> in the
existing one). Fixed by merging it in as a <para>, verified green in a
follow-up commit (4195e82).

What I ran

  • dotnet build on PerformanceMonitor.Analysis, Darling/Darling.Tests,
    Lite/PerformanceMonitorLite.csproj, Lite.Tests/Lite.Tests.csproj with
    -p:EnableWindowsTargeting=true — all green, 0 warnings, 0 errors, after
    the doc-comment fix.
  • Targeted classes on a live TimescaleDB rig (timescale/timescaledb:2.28.1-pg18,
    DARLING_TEST_PG set): RobustBaselineTests (35), LocalClockBucketKeyTests
    (13), AnomalyGateNullWindowMonteCarloTests (11), AnomalyGateTilesTests
    (17), PgTargetAnomalyTests (44), DarlingAnomalyBaselineTests (18),
    DarlingAnalysisStoreTests (9), DarlingAnalysisBudgetTests (3),
    DarlingAnalysisServiceEngineRoutingTests (11),
    DarlingAnalysisPipelineTests (11) — all green, 0 failed, 0 skipped.
  • AnomalyNullWindowLiveTests (still [Fact(Explicit = true)], HistoryDays = 10), run with -explicit on -showLiveOutput on both origin/dev and this
    branch against the same live rig:
    • dev: DIAG shiftFires4h=0 shiftPasses4h=3 shiftFires24h=0 shiftPasses24h=3 spikeAlone4h=0 spikeAlone24h=0 null4h=0/57 null24h=0/12
    • this branch: identical DIAG line, byte for byte.
      Honest read: no change at 10 days of history for this particular fixture's
      pattern — its z-path outcome here isn't gated by tier trust in the way that
      moved. I'm reporting the DIAG lines as instructed rather than reading a
      result into them.
  • Full Darling suite once, DARLING_TEST_PG unset:
    Total: 13671, Errors: 0, Failed: 214, Skipped: 730, Not Run: 2. Diffed
    the failing-test names byte for byte against the same run on origin/dev
    (Total: 13665, Failed: 214) — identical failing-test set, all
    pre-existing WPF/Windows-only tests that don't run on this Mac. This
    branch adds 6 new tests (the ones above) with zero new failures and zero
    regressions.
  • Lite/PerformanceMonitorLite.csproj and Lite.Tests/Lite.Tests.csproj
    built clean with -p:EnableWindowsTargeting=true. Did not run
    Lite.Tests (hangs on this Mac per brief); left to CI.

Watch in CI (Lite.Tests, 3–20 day seeded-history fixtures asserting the absolute path)

  • AnomalyDetectorTests.DetectCpuAnomalies_ThinBaseline_AbsoluteFallbackFires_NotSilent
    and AnomalyDetectorTests.DetectCpuAnomalies_ThinBaseline_SuppressesPureZSpikeBelowAbsoluteBar
    — both use SeedThinBaselineCpu (2 distinct days, 16 samples). By this
    lane's rule this Full bucket is untrustworthy, and there's no HourOnly or
    Flat sentinel/pooled data seeded in that fixture, so both stay on the
    absolute-fallback path exactly as asserted. Flagging anyway since they're
    exactly the shape this lane changes behavior for.
  • AnomalyDetectorTests also has SeedThinBaselineWaitRates (2 distinct
    days) feeding wait-rate anomaly tests — same reasoning, same expected
    outcome (no seeded coarser tier to fall back to, so unchanged).
  • ComparisonBandingTests.AnUntrustworthyBaseline_FallsBackToTheAbsoluteRule
    constructs a single untrustworthy BaselineBucket directly (not through
    SelectBucket/a map) and asserts on ComparisonBanding, not on tier
    selection — unaffected by this change, but flagging since it lives in the
    same 1-distinct-day territory.
  • I did not find any Lite.Tests fixture with 3–20 days of history where a
    coarser sentinel/pooled tier IS present and trustworthy, so I did not
    expect (and did not find, by build) any Lite.Tests compile-time
    expectation change.

Not verified

  • Live behavior against a real newly-onboarded target end to end (only unit-
    level SelectBucket fixtures and the one AnomalyNullWindowLiveTests
    demonstration, which showed no delta at 10 days for its specific fixture).
  • Lite.Tests actual pass/fail (build-only on this Mac; CI runs it).

CHANGELOG entry

On a newly monitored server, anomaly detectors use the hour-of-day or overall baseline while the hour-of-week one is still too young to trust, instead of only the absolute threshold. Anomaly detection now starts in days rather than about three weeks (#3653).

…y tier

A newly monitored target's Full (hour+dow) bucket clears its sample-count
floor on day one but not its 3-distinct-day floor for about 3 weeks, so
every z-score detector was dark until then even when the coarser HourOnly
or Flat tier was already trustworthy.

SelectBucket now computes today's tier-by-sample-count choice unchanged,
returns it immediately when it is trustworthy or zero-history, and
otherwise walks only the coarser tiers (built exactly as today, sentinel
or pooled) for the first one that is IsTrustworthy. If none is, returns
the original chosen bucket unchanged (today's absolute-fallback path).

Adds 6 RobustBaselineTests covering: exact-sentinel HourOnly fallback,
fallback through to Flat, all-untrustworthy (absolute fallback preserved),
IsZeroHistory never overridden, trustworthy Full unchanged (same instance),
and pooled-synthesis fallback with no sentinels.
DocCommentHygieneTests caught two stacked <summary> openings on the same
member. Merged the #3653 addition into the existing summary as another
<para> block, same content.
@erikdarlingdata
erikdarlingdata marked this pull request as ready for review September 24, 2026 18:35
@erikdarlingdata
erikdarlingdata enabled auto-merge (squash) September 24, 2026 18:36
BaselineBucket.IsYoung is true only for the untrustworthy-for-too-few-
distinct-days case the tier walk exists for: real dispersion and enough
samples, but too few DISTINCT days for this tier. Never true for a
trustworthy bucket, a zero-dispersion bucket (constant-valued fixtures,
idle metrics), or a thin (too-few-sample) bucket.

BaselineMath.SelectBucket now walks coarser only when !chosen.IsYoung
fails to hold, i.e. only for young buckets, instead of walking on
!IsTrustworthy && !IsZeroHistory (which also fired on zero-dispersion
and thin buckets and broke 3 Lite tier-selection tests in CI).

Adds two RobustBaselineTests: a zero-dispersion Full bucket with a
trustworthy coarser tier present (not walked, returns Full), and a
boundary case pinning that the walk fires on distinct-days alone, not
on a sample-count deficit (a genuinely thin Full bucket is never
selected as Full in the first place, so IsYoung's sample floor can
only be observed at the floor's boundary).
…stinct days

Each failing fixture's chosen bucket was IsYoung under the narrowed
tier walk (real dispersion, enough samples, too few distinct days for
its tier), so the walk correctly moved it coarser to Flat — a
by-design change, not a bug. Rather than flip the expected tier, give
each fixture enough distinct days to keep pinning what it was written
for:

- BaselineProviderTests.GetBaseline_SparseBucket_CollapsesToHourOnly:
  9 further distinct days added so the pooled hour-only tier clears
  its own 10-day floor and is not itself walked past to Flat.
- BaselineProviderTests.GetBaseline_HourOnly_SumsDistinctDaysAcrossDowBuckets:
  4 distinct days -> 10, for the same reason (still pins the SUM
  behavior across dow buckets, at the day count HourOnly's own floor
  requires).
- LocalClockBucketKeyTests.Provider_ZoneId_EstAndEdtRowsAcrossTheSpringForward_ShareOneLocalBucket:
  a third distinct Tuesday added to each of the two DST-straddle
  buckets so DistinctDays clears Full's 3-day floor; still pins the
  EST/EDT keying and the rollover dating, unchanged.
@erikdarlingdata

Copy link
Copy Markdown
Owner Author

Lane L1e-2 done. Narrowed the tier walk to YOUNG buckets only, per the ruling.

Changes (91a978e, 9403808 on feat/3653-b-trusted-tier):

  • BaselineBucket.cs: added IsYoung — EffectiveStdDev > 0 && SampleCount >= sampleMin && DistinctDays < dayMin — true only for the untrustworthy-for-too-few-distinct-days case, never for a trustworthy, zero-dispersion, or thin bucket.
  • BaselineMath.SelectBucket: replaced if (chosen.IsTrustworthy || chosen.IsZeroHistory) return chosen; with if (!chosen.IsYoung) return chosen;. Doc comment updated to explain why the old !IsTrustworthy gate over-walked.
  • RobustBaselineTests.cs: added (g) a zero-dispersion Full bucket with a trustworthy coarser tier present — not walked, returns Full; and (h) a boundary test — since CollapseThreshold (10) equals Full's own trust sample floor, a genuinely sub-floor-sample Full bucket is never selected as Full in the first place (falls through to HourOnly/Flat in SelectBucketBySampleCount), so a literal "thin Full bucket in the 10-14 hysteresis band" per the brief's (h) isn't reachable from the chosen-Full arm. Pinned the boundary instead: a Full bucket sitting exactly at the sample floor with too few distinct days is walked on DAYS alone.
  • Gave the 3 CI-failing Lite fixtures (BaselineProviderTests.GetBaseline_SparseBucket_CollapsesToHourOnly, BaselineProviderTests.GetBaseline_HourOnly_SumsDistinctDaysAcrossDowBuckets, LocalClockBucketKeyTests.Provider_ZoneId_EstAndEdtRowsAcrossTheSpringForward_ShareOneLocalBucket) enough distinct days to clear their tier's own floor — all three were IsYoung under the narrowed rule (by design), so the fix was giving them more days, not flipping the expected tier.

Verified: Darling build (-p:EnableWindowsTargeting=true) green. Targeted runs green: RobustBaselineTests 37/37, all *Baseline* classes, AnomalyGateTilesTests 17/17, AnomalyGateNullWindowMonteCarloTests 11/11 — no [FAIL]. Lite.Tests builds clean; execution left to CI per policy (WPF host hangs on this Mac). Round-1 CI run 36044101657 failed on exactly the 3 named Lite tests plus a live PG DST test unrelated to this fix (pre-existing on this branch, not touched). Round-2 CI (this push) not yet observed by me — handing to pr-tender to poll and confirm green within the 2-round budget.

Not verified: full Darling suite with DARLING_TEST_PG unset (214 baseline) — ran out of budget window; targeted classes above cover the touched surface. No live PG rig was started for this lane.

Coordinator's diagnosis: this test pins DST KEYING (which bucket the
rows land in), not tier selection, and the young-target walk (this PR)
correctly moves both the CPU and the rollover Full bucket to Flat --
CPU's bucket has real spread across two Tuesdays (DistinctDays 2 <
Full's day floor 3); the rollover bucket clears IsYoung too, not from
real spread (it's ten identical 99s) but because CPU's 5.0 absolute
dispersion floor keeps EffectiveStdDev above zero regardless. Both
walks fall through HourOnly (equally thin) to the same Flat bucket,
which pools all 22 samples across three distinct Tuesdays and clears
its 3-day floor.

Read the RAW Full bucket through GetBucketMapAsync's dictionary
(map.Buckets[(hour, dow)]) for the keying assertions the test actually
cares about, and added separate assertions pinning the new walk
behaviour (IsYoung on both raw buckets; Tier == Flat, IsTrustworthy,
and the pooled SampleCount on the walked GetBaselineAsync result).
@erikdarlingdata
erikdarlingdata merged commit 1705f4f into dev Sep 24, 2026
15 of 16 checks passed
@erikdarlingdata
erikdarlingdata deleted the feat/3653-b-trusted-tier branch September 24, 2026 19:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant