Skip to content

PostgreSQL wait standouts, the I/O admission floor and the ratio multiple say what the second fleet read found: measured per family, cited 2026-09-20, and every Aurora wait fact stops calling a validated bar a guess (#3691 lineage flip, round 2) - #3785

Merged
erikdarlingdata merged 1 commit into
devfrom
fix/3691-lineage-measured-2
Sep 20, 2026

Conversation

@erikdarlingdata

Copy link
Copy Markdown
Owner

What does this PR do?

The PostgreSQL bars the second fleet calibration read (batch C, 2026-09-20) now say measured, cite that read per family, and every Aurora wait fact carries threshold_lineage = 1 — partial for #3691, the checklist line Lineage flip, round 2 (calibration C). No bar value changes. The only non-comment code in the diff is the wait scorer's stamp predicate; everything else is what the code SAYS about its numbers and the flag get_analysis_facts shows.

The lie this stops telling

Round 1 (#3723) flipped what the 2026-09-19 read of LEVELS could support and was honest about the rest: the wait STANDOUT bars stayed unmeasured because that read aggregated by type; the I/O quarter-hour admission floor stayed unmeasured because §B1 distributed reads per HOUR; the ratio families' firing multiple stayed unmeasured because nobody had read hour-of-week ratios at all. On 2026-09-20 the second read (the same 50 Aurora PostgreSQL clusters) went per event, per quarter-hour, and per ratio — and those three comments are now the other lie: a standout fact at Lock:relation 0.15 is telling an operator "this bar is a judgment" when it is ≈ p99.6 of that event's buckets.

What flips (values in parentheses unchanged)

Bar Read (2026-09-20, 50 Aurora PostgreSQL clusters) Comment Flag
WaitStandoutConcerning (0.15) / WaitStandoutCritical (1.0) — PgTargetScorer.Waits.cs §C3, per event, 7 d: for non-IO events 0.15 ≈ p99.6 and 1.0 ≈ p99.99 of 5-min buckets (Lock:relation max 2.01 once; LWLock:WALWrite / IO:WALSync never near). Stated honestly for IO:DataFileRead: 3.7 % of its buckets ≥ 0.15 (≈ p96.3), 0.25 % ≥ 1.0 — the same band the IO rollup's own 0.40 ≈ p99 occupies; reading is what a database does measured standouts 0 → 1; yielded rollups 0 → 1 (the deciding bar is now measured); sampled stay 0
IoBaselineBucketMinimumReads (250) — PgTargetScorer.Io.cs §C5, 14 d: 57 % of fleet 15-min buckets carry < 250 reads (62 % < 1,000); per server 0–100 % (median 70 %); 25/50 have a MEDIAN of 0 reads per quarter-hour. Consequence stated on the constant: at ≤ 17 samples per 30-d hour-of-week bucket, excluding 57–100 % leaves half the fleet under the 15-sample restore threshold → their I/O baseline is the hour-of-DAY collapse (0.85) by construction; hour-of-week only on the ~25 % that read continuously measured (as an ADMISSION floor, not a grading bar) I/O anomaly was already 1; verified the stamp and the bucket_reads_floor metadata still say so
AnomalyThresholds.PgRatioAnomalyThreshold (3.0) §C2, 28 d (centres from 21, ratios over 7; 100,800 samples), per family: TPS ratio p99 1.32 / p99.9 1.65 / p99.99 24.4, share ≥ 3 = 0.024 % → ≈ p99.97, well placed. Wait rate p99 12.9 / p99.9 31, share ≥ 3 = 7.1 % → routine ALONE (≈ p93); what decides is PgWaitProfileFallbackMsPerSec (500 = 8× the fleet p99 0.062 ms/ms) and #3780's peak-AND-mean gate. Deadlocks/h: only 22 of 8,400 (server, bucket) pairs have a non-zero centre; no ratio ≥ 3 → an empty interval on a population too thin to place anything; the is_new arm fires in practice measured, with the three verdicts written on the constant as a list see below — no detector flag moves

The detector flags, checked bar by bar (the brief asked me to state this)

  • TPS / CPU anomalies: already 1 since PostgreSQL bars the fleet measured now say so: lineage comments cite the 2026-09-19 read and threshold_lineage flips to 1, so a validated bar stops calling itself a guess (#3691 lineage flip) #3723 — they are Z-DETECTORS on PgTpsFloor/PgTpsFallback and PgCpuFloorPct/PgCpuFallbackPct and never grade on the ratio multiple, so §C2's TPS verdict is recorded on the constant "for the record" and flips nothing. CPU ratios were not separately measured and did not need to be: no CPU arm reads the multiple.
  • Wait-profile anomaly: stays 0. Its robust arm is decided on HeavyTailModifiedZThreshold — the SQL Server fleet's cutoff by reference, unmeasured for PostgreSQL — and PgTargetScorer.ScoreRatioAnomaly stamps 0 on every ratio family because its ramp spans (RatioAnomalySaturation 9×, WaitProfileModifiedZSpan, FirstOccurrenceExceedanceSpan) are chosen. The stamp comment now says the multiple was READ and is routine alone, and names the floor + gate as what decides.
  • Deadlock-rate anomaly: stays 0, as the brief directed and as the read supports — 22 pairs cannot PLACE a multiple (the flag contract needs a placed bar, not a word), and the scorer's saturation span is chosen. The stamp says exactly this.
  • Sampled-wait bars (PgSampledWaitProfileFallbackMsPerSec, WaitSampledMinimumSamples): stay unmeasured, now with the note "Population 0 on the dogfood fleet as of 2026-09-20 (no pg_wait_sampling cluster)" so nobody re-reads an empty table for them (§C6).

Why this shape

  • The wait stamp is decided by POPULATION, once, for both exits. Round 1's predicate was "rollup AND Aurora"; with the standout bars measured, every row in s_waitBars is measured on the Aurora population and the honest verdict is the instrument's: Aurora deltas → 1, stock sampled estimate → 0. The yielded-rollup exit used to write a bare 0 ("decided on the unmeasured standout bar"); it now writes the same population verdict, because the bar that decided it is measured. One variable, two stamps, no way for the two exits to disagree.
  • A "measured" constant whose family stays at 0 is stated per family on the constant. PgRatioAnomalyThreshold's comment is a three-item list because the same number has three different verdicts, and a single adjective would be wrong for two of them.
  • Not moved: RatioAnomalySaturation. Nobody has asked where a ratio family should saturate; §C2 gives the tails (TPS p99.99 24.4, wait p99.9 31) that a future read would calibrate the ramp's top against, and the comment now says so instead of "calibrate the multiple", which is done.

Falsified doc lines fixed (in-lane, smallest diff)

PgTargetAnomalyDetector.cs class remark + wait-profile summary + the classical-arm inline (unmeasured: → measured: with the two dates); PgTargetAnomalyDetector.Io.cs inline "floor — unmeasured" → measured §C5; PgTargetScorer.Anomaly.cs RatioAnomalySaturation summary + the ramp's inline marker; AnomalyThresholds.cs lane-9 block paragraph ("the ratio families' firing multiple is unmeasured" was about to be false). Not touched: PgTargetScorer.Queries.cs (C4 says the share bars are routine given busy — a ruling is filed on the face; lineage stays as lane L set it), the Sessions files (lane 25 is in them), any SQL, any shared switch.

Proof no bar moved

git diff origin/dev -U0 -- PerformanceMonitor.Analysis Darling/PerformanceMonitor.Darling.Analysis \
  | grep -E "^[-+]\s*(public|private|internal) const" | wc -l   # → 0

The complete non-comment code diff in the two analysis assemblies:

+        var isMeasuredPopulation = fact.Metadata.GetValueOrDefault(WaitIsSampledKey) <= 0;
-            fact.Metadata["threshold_lineage"] = 0;
+            fact.Metadata["threshold_lineage"] = isMeasuredPopulation ? 1 : 0;
-        var isMeasuredRollup = fact.Metadata.GetValueOrDefault(WaitIsStandoutKey) <= 0
-        fact.Metadata["threshold_lineage"] = isMeasuredRollup ? 1 : 0;
+        fact.Metadata["threshold_lineage"] = isMeasuredPopulation ? 1 : 0;

Blast radius

Analysis-assembly comments; one predicate in the wait scorer. Every fired Aurora wait STANDOUT (Lock:relation, LWLock:WALWrite, IO:DataFileRead, IO:WALSync) and every rollup that yields to one shows threshold_lineage: 1 on the next pass; stock sampled facts unchanged at 0. No SQL, collector, shared switch or migration; a SQL Server pass is untouched (no shared file edited).

Which component(s) does this affect?

  • Lite
  • Darling
  • Lite Tests
  • Darling Tests
  • SQL collection scripts
  • Documentation
  • Full Dashboard (deprecated)
  • CLI Installer (deprecated)

How was this tested?

Changed pins (PgTargetMeasuredLineageTests, extended not forked): s_measured entries now carry the DATE their block must cite (CalibrationDate 2026-09-19 for round 1, SecondCalibrationDate 2026-09-20 for WaitStandout*, IoBaselineBucketMinimumReads, PgRatioAnomalyThreshold); those three left s_stillUnmeasured; RatioAnomalySaturation / WaitProfileModifiedZSpan joined it; a new s_sampledPopulationZero roster pins the population-0 note on both sampled bars; the Waits flag pin is now "exactly two isMeasuredPopulation ? 1 : 0 stamps, no bare = 0; or = 1;". PgTargetWaitTests: standout / below-bar standout / yielded rollup 0 → 1 with the reason; sampled pins unchanged at 0. PgTargetWaitLiveTests: every Aurora fact 1 (was: only the LWLock rollup), stock still 0. PgTargetIoTests: the floor's summary must say <b>measured</b> as an admission floor, cite 2026-09-20, the 57 % figure and the hour-of-DAY consequence, and no longer say unmeasured. PgTargetAnomalyTests: the per-detector verdict comment restated (values unchanged: TPS/CPU 1, sessions/deadlock 0).

Executed locally (net10.0 harness at obj/mactest/, AssemblyName=Darling.Tests, deleted before commit): PgTargetMeasuredLineageTests, PgTargetThresholdLineageTests, PgTargetWaitTests, PgTargetIoTests, PgTargetAnomalyTests, PgTargetSampledWaitTests, PgTargetWaitLiveTests — 93/93 green, 0 skipped, the live classes against a throwaway timescale/timescaledb:2.28.1-pg18 on desktop-linux (torn down). Censuses: DocCommentHygieneTests, FleetIdentifierScrubTests, CommentFilterAdoptionTests, RepoFileAdoptionTests, PgTargetSharedSwitchRoutingTests, PgTargetFactCollectorTests — 120 green, 1 skipped (live).

Builds (zero new warnings): PerformanceMonitor.Analysis, PerformanceMonitor.Darling.Analysis, Darling/Darling.Tests, Lite.Tests, deprecated/Dashboard.Tests (-p:EnableWindowsTargeting=true). Pre-existing warnings untouched: ConfigChangeTraceAnchorTests.cs(94) xUnit2013, DefaultTraceEventsCollectorDefinitionTests.cs(211) xUnit2013, LogTailOverlapThresholdPinTests.cs(95) xUnit2000.

CI's Windows job is the first execution of everything not in the harness set.

Checklist

  • I have read the contributing guide
  • My code builds with zero warnings (dotnet build -c Debug)
  • I have tested my changes against at least one SQL Server version — n/a: PostgreSQL-target analysis only; the SQL Server pass is untouched (no shared file edited)
  • I have not introduced any hardcoded credentials or server names

Partial for #3691 — checklist line: Lineage flip, round 2 (calibration C). Do not close.

…he wait standout bars, the I/O admission floor and the ratio multiple cite the 2026-09-20 read per family, and every Aurora wait fact now carries threshold_lineage = 1 (partial #3691)

No bar value changes. The wait scorer's stamp is decided by population (Aurora deltas vs stock sampled estimate) on both its exits, since the standout bar that decides a yield is measured too. The deadlock-rate and wait-profile anomalies keep 0 and their stamps say why (unplaced multiple on 22 pairs; SQL Server heavy-tail cutoff by reference; chosen ramp spans). Sampled-wait bars gain the population-0 note (§C6).
@erikdarlingdata
erikdarlingdata merged commit 9b41f94 into dev Sep 20, 2026
6 of 7 checks passed
@erikdarlingdata
erikdarlingdata deleted the fix/3691-lineage-measured-2 branch September 20, 2026 13:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant