The write family tells Aurora the truth: checkpoint pressure and max_wal_size are not applicable where storage owns checkpointing, and WAL volume is graded against its own baseline where the engine reports it (partial #3691)

[erikdarlingdata]

What does this PR do?

Headline mechanism. Lane 15 of #3691 (design §3.11; calibration §A4 / §A7) makes the PostgreSQL-target write family tell Aurora the truth, in two halves. On aurora-postgres, PG_CHECKPOINT_PRESSURE and CONFIG_PG_MAX_WAL_SIZE are not_applicable — emitted with the shape the operator can see and the reason, scored 0, rooting nothing. Where the engine reports WAL, PG_WAL_VOLUME_SHIFT states the window's WAL volume and ANOMALY_PG_WAL_VOLUME grades it against the server's own hour-of-week baseline, lifting and folding onto the checkpoint incident it leads. PARTIAL for #3691 — do not close. Checklist lines completed verbatim: Aurora-suppress PG_CHECKPOINT_PRESSURE and CONFIG_PG_MAX_WAL_SIZE and PG_WAL_VOLUME_SHIFT.

Half one — what fifty Aurora clusters measured, and what the product said about it

The calibration pass (§A4) read fourteen days of pg_write_stats on every PostgreSQL target the fleet monitors — fifty Aurora clusters — and found the same shape on all fifty: sixty timed "checkpoints" an hour (one a minute), a requested share whose p99 was 0 and whose maximum was 0.033. That is not fifty well-tuned servers. Aurora's storage layer owns checkpointing; the checkpointer counters the engine surfaces are synthetic, and max_wal_size governs nothing there. So v1's PG_CHECKPOINT_PRESSURE was a permanent, measured 0 that meant nothing, and CONFIG_PG_MAX_WAL_SIZE at its shipped default was a convention finding about a knob the engine never consults — a 0.4 advisory on every Aurora target in the fleet, forever, for a setting nothing reads.

The mechanism. The registry fact's is_aurora (MonitoredEngineKind, never a column's presence — #2530; the same read the buffer composite takes) gates two stamps:

• The collector (PgTargetFactCollector.Write.cs) still emits the pressure fact — the operator should see WHY — and stamps not_applicable = 1, not_applicable_on_aurora = 1, reason_aurora_storage_checkpointing = 1, plus timed_per_hour beside the existing requested_share. The config collector stamps the knob fact not_applicable = 1 / not_applicable_on_aurora = 1 (the fact is still emitted: the write partial reads checkpoint_timeout_s and bytes off it, and the value is true).
• The scorers (PgTargetScorer.Write.cs, .Config.cs) return 0 off the flag before any bar is consulted, and take no lineage stamp — there is nothing graded, so there is no bar whose lineage the fact should disclose.
• Base 0 makes everything else follow without a second gate anywhere: FactScorer skips amplifiers at base 0 (the knob's co-fire can never lift it), the pressure → knob edge reads BaseSeverity > 0 and is false, InferenceEngine roots nothing at 0. An Aurora-stamped server with planted requested-DOMINANT checkpoints — two requested and one timed a minute, a shape Aurora cannot produce — forms no checkpoint story and no max_wal_size advisory (pinned, live).
• The advice on a flagged fact is one block: what Aurora does instead (storage owns checkpointing; the counters are synthetic — the fact's own timed_per_hour and share are stated), and where to look (get_pg_io_stats, the CPU and wait findings; get_pg_write_stats shows the synthetic series). No sizing arithmetic — there is no knob to turn.

Why metadata flags and not a status. Fact.Metadata is Dictionary<string, double> and the pipeline has no fact-status vocabulary; the existing idiom for "this arm is structurally absent on this flavour" is a numeric flag whose KEY is the meaning (hit_ratio_suppressed, wal_tracked, evictions_tracked). "Status" and "reason" are therefore flag keys — not_applicable, reason_aurora_storage_checkpointing, and on the shift fact unavailable / reason_wal_stats_not_reported / reason_pg_stat_wal_absent — rendered by get_analysis_facts exactly as every other flag is. Inventing a parallel status field was considered and rejected: it would touch the shared model and every renderer for one family's benefit.

Stock PostgreSQL is byte-identical. Lane 2's pins (PgTargetKnobsTests) pass unchanged; PgTargetWriteTests re-runs the same facts with and without the flag side by side.

Half two — PG_WAL_VOLUME_SHIFT states, ANOMALY_PG_WAL_VOLUME grades

§A7: wal_bytes was 0 on all fifty clusters for the whole span — Aurora does not populate pg_stat_wal (the collector types the WAL columns NULL there; pg_stat_get_wal() raises 0A000). So a WAL-volume fact on this fleet would be a fabricated 0, and the shift is built for the engines that report it, with the honest unavailable where they do not.

One differencing, three readers. A new read, PgTargetFactCollector.PgTargetWalVolumeSql, rates Δwal_bytes per collection over the collection's own gap — the checkpoint read's reset-clamped differencing (LAG per sample, GREATEST(raw, 0), wal_stats_reset IS DISTINCT FROM its LAG), the pg_tps arm's second-truncated LAG(collection_time) gap — and returns the window's peak, mean, rated-sample count, total bytes, wal_records, reset count and trackedness. The detector's window read is this text by alias (PgTargetAnomalyDetector.WalVolumeWindowSql), and the new pg_wal_bytes_per_sec baseline arm (PgTargetBaselineProvider.Wal.cs) is the same sampled/clean with the half-open upper bound, wal_bytes IS NOT NULL so an untracked series yields an EMPTY bucket rather than thirty days of zeros with a zero MAD, under the one RobustTierScaffold. The fact's mean and peak, the detector's peak and the bucket are one unit — bytes of WAL per second per collection.

The shift is CONTEXT. PG_WAL_VOLUME_SHIFT: Value = the window's mean bytes/s, peak_wal_bytes_per_sec beside it, base 0 always, threshold_lineage = 1 — not "measured" but "no bar was chosen to have a lineage": WAL volume is a server-relative quantity (9 MiB/s is routine on one server and a deployment gone wrong on another), so this family draws no absolute bar for it anywhere. Trackedness is read three ways because the store has three ways of saying "not reported": all-NULL wal_bytes (Aurora, pre-14) → unavailable = 1, reason_wal_stats_not_reported; zero bytes AND zero records across a window in which checkpoints ran → the same reason (a live pg_stat_wal writes a record per commit); a registry major below 14 with no rows at all → reason_pg_stat_wal_absent. A known 14+ major with no rows emits nothing — the coverage witness already describes that gap.

The anomaly is the graded instrument. DetectWalVolumeAnomalies (PgTargetAnomalyDetector.Wal.cs) judges the window's peak against the pg_wal_bytes_per_sec bucket through the shared gate — robust modified-z at 3.5 with median/MAD, classical 2σ otherwise — with PgWalBytesFloorPerSec = 1 MiB/s as the magnitude floor and PgWalBytesFallbackPerSec = 16 MiB/s as the bar on an untrustworthy bucket (16 MiB/s fills the shipped 1 GB max_wal_size in about a minute — the rate at which a default-configured server is already checkpointing on volume). Both are unmeasured, and stated as un-calibratable on the current fleet: no PostgreSQL target the product monitors reports WAL, so the fact carries threshold_lineage = 0 until a stock population is measured. The detector reads the four-hour window FIRST and returns on an untracked window without asking for the thirty-day bucket — the reverse of the five v1 detectors, on purpose: on this fleet every pass would otherwise run the family's largest read for nothing. baseline_ratio is stamped as peak over the bucket's robust centre (median when it has one, else mean — the centre the deviation prose names, so one paragraph's two numbers are one number).

How the anomaly reaches the checkpoint story — and why there is no graph edge. Two relationships, both already where the shared pipeline puts anomaly → parent relationships: PgTargetFactKeys.AnomalyToFamilies folds the anomaly's story onto the checkpoint incident (P2 routed it), and the pressure fact's trigger amplifier now reads ANOMALY_PG_WAL_VOLUME fired — the anomaly, not the context fact, is the thing that can have fired (the anomaly source is in ScoreAll's lookup; base 0 on Aurora means the amplifier pass never runs there, and the detector emits nothing where WAL is unreported). Lane 2's inert PG_WAL_VOLUME_SHIFT → PG_CHECKPOINT_PRESSURE edge is removed: a base-0 fact can neither root a story nor be walked to, so the edge was dead by construction and its doc line had become false. Re-keying it onto the anomaly — which the STEP brief sketched — was considered and rejected, with the argument recorded in PgTargetRelationshipGraph.Write.cs: InferenceEngine roots by descending severity and consumes the walked path, so on any pass where the anomaly outranked the pressure fact the persisted finding would root on ANOMALY_PG_WAL_VOLUME with the pressure as its leaf — capped at the anomaly's 1.49 tuning-class ceiling however hard the pressure paged, and flapping its root key from pass to pass. No v1 anomaly has a graph edge for the same reason. The live pin shows the intended shape instead: pressure at share 1.0 × (1 + 0.3 knob + 0.2 trigger) = exactly 1.5, the trigger carrying it to the notify floor, the anomaly card folded onto that incident.

Advice. The shift's block states the measured mean, peak and total (or the reason it cannot, and where per-statement WAL still lives — get_pg_top_queries). The anomaly's block is the deviation family's (ComposeDeviation: peak, sigmas, centre, sample count; first-occurrence wording on a thin bucket) plus the ratio to the routine rate ("8.76× the 1.1 MB/s this server routinely writes at this hour of the week"), the deployment-correlation framing (a shift is a deploy, a batch, a bulk load or a reindex before it is a setting), whether checkpoint pressure co-fired, and the levers with their costs: wal_compression trades CPU for bytes; max_wal_size absorbs volume at the cost of pg_wal footprint and recovery time; batch sizing is the application's. full_page_writes is posture and appears nowhere (pinned). The static block is built on first use, not as a static readonly initializer — it concatenates s_anomalyHedge from another partial file, and static-field order across partial files is unspecified (the aliasing trap the wave-2 brief records).

Every file, one line

File	Why
Darling/…/PgTargetFactCollector.Write.cs	PgTargetWalVolumeSql; the Aurora not_applicable stamp; PG_WAL_VOLUME_SHIFT in its tracked and unavailable shapes; body split into ReadCheckpointPressureAsync / ReadWalVolumeAsync (not Collect*Async — the collect-surface census reflects those)
Darling/…/PgTargetFactCollector.Config.cs	the knob's not_applicable stamp off the registry fact
PerformanceMonitor.Analysis/PgTargetScorer.Write.cs	0 off the flag, no lineage stamp; trigger amplifier re-pointed to the anomaly; class doc
PerformanceMonitor.Analysis/PgTargetScorer.Config.cs	the max_wal_size arm returns 0 off the flag
PerformanceMonitor.Analysis/PgTargetRelationshipGraph.Write.cs	the dead shift edge removed; the argument against an anomaly edge
PerformanceMonitor.Analysis/PgTargetAdvice.Write.cs	Aurora one-sentence block; shift composer (tracked / unavailable); anomaly composer + lazily-built static
PerformanceMonitor.Analysis/PgTargetAdvice.Anomaly.cs	boundary extension: one delegating case to the WAL composer, and the summary line it falsified
PerformanceMonitor.Analysis/Baselines/AnomalyThresholds.cs	the // lane 15 block: PgWalBytesFloorPerSec, PgWalBytesFallbackPerSec, each with its unmeasured lineage
Darling/…/PgTargetAnomalyDetector.Wal.cs	WalVolumeWindowSql alias; the detector body (window first, bucket second)
Darling/…/PgTargetBaselineProvider.Wal.cs	the pg_wal_bytes_per_sec arm
Darling/Darling.Tests/PgTargetWriteTests.cs	NEW: 19 unit pins + two DARLING_TEST_PG e2es (below)
Darling/Darling.Tests/PgTargetAnomalyTests.cs	censuses re-pinned: served metrics 5 → 6 (pg_wal_bytes_per_sec, pg_write_stats), detector SQL roster + alias pin, AnomalyThresholds name list + bars in bytes, stub list minus lane 15
Darling/Darling.Tests/PgTargetKnobsTests.cs	the shift-edge pin → no edge into pressure (lane 12's shift → PG_SLOT_RETENTION edge stays pinned by membership); the churn e2e now reads two write facts and pins the shift's planted rate

Lite twin: none — every touched class is Darling-only or the shared PerformanceMonitor.Analysis (no Lite copy). deprecated/Dashboard.Tests builds clean.

What it does NOT do

No rung; no new key, source or metric name (all P2's); no root-file edit (PgTargetFactKeys.WalVolumeShift's doc line "declared for the write-chain edge" and PgTargetBaselineProvider.cs's retention table list are now stale — reported for the coordinator's shared-file batch). Lane 12's PG_WAL_VOLUME_SHIFT → PG_SLOT_RETENTION edge (#3727) is left in place: the same base-0 argument makes it inert, and it is lane 12's file — reported on #3691 for the coordinator, not removed here. No measured WAL bar — none exists to measure. The stock-target 1.30 anomaly ceiling stays a v3 item. A SQL Server target's pass is byte-identical: every new arm is behind a pg_-source switch or a PG-only key.

Test plan

Builds, zero warnings of ours: PerformanceMonitor.Analysis, PerformanceMonitor.Darling.Service (pulls Darling.Analysis), Darling.Tests, Lite.Tests, deprecated/Dashboard.Tests (-p:EnableWindowsTargeting=true on the test projects). The one Lite.Tests warning (LogTailOverlapThresholdPinTests.cs:95 xUnit2000) is pre-existing on dev.

New tests (PgTargetWriteTests, 21): scorer 0 off the flag with no lineage stamp and stock at 1.0 with it; max_wal_size 0.4 / 0 by flag (theory); an Aurora fact set with requested-dominant pressure → no amplifier ran, no active edge, no write-chain story, shared_buffers still roots; the same set on stock → pressure → knob (lane 2's contrast); the shift scores 0 in all three shapes with lineage 1 and no amplifier through ScoreAll; the trigger amplifier ignores the context fact and matches the anomaly (0.625 × 1.2 pinned); fold + no edge from either key + no edge INTO either; the gate fires at 9.2 and 2 MiB/s, holds under the 1 MiB/s floor at 20+σ, falls back to 16 MiB/s on a thin bucket; the read's differencing, dialect and alias; the baseline arm's shape; Aurora advice (synthetic shape stated, no sizing), shift advice (rates or reason), anomaly advice (peak, σ, ratio, deploy framing, co-fire sentence, first-occurrence wording, never full_page_writes, FactAdvice delegation equality). Gated e2es: Aurora-stamped, two requested + one timed a minute, WAL NULL → one finding (shared_buffers), pressure fact not_applicable with timed_per_hour 60 and share ⅔, shift unavailable / reason_wal_stats_not_reported, knob 0 with the flag; stock-stamped, 31 days at 1 MiB/s (±5 % ripple) then 12 MiB/s for the window, requested-dominant throughout → trusted bucket at the median, anomaly at exactly 12 MiB/s with lineage 0 and ratio in [11, 13], pressure story at 1.5 with the trigger carrying it, anomaly folded onto its incident, shift stating 12 MiB/s, no card on the shift.

EXECUTED here through a net10.0 harness (AssemblyName=Darling.Tests, the 40-file PgTarget census set) run from inside the worktree against a throwaway timescale/timescaledb:2.28.1-pg18 on Docker context desktop-linux, migrated to V132 by the fixture, torn down after: 561 / 562 green after the rebase onto dev at 50881f8c5 (the set plus lanes 11 / 12 / 13 / 5's PgTargetIoTests, PgTargetReplicationTests, PgTargetBloatTests, PgTargetWaitTests), the one red being the harness csproj's own presence tripping DocCommentHygieneTests.TheDerivedProjectListCoversEveryProjectInTheTree (the artifact #3665 and #3715 recorded). That includes both new e2es and every existing gated e2e in the set (lane 2's churn e2e with its re-pinned two write facts; lane 9's spike e2e). Every number asserted was read off the executed run. First execution of everything outside the set is CI's Windows job.

Censuses pre-checked: PgTargetFactCollectorTests (collect surface unchanged — the two new helpers are Read*, not Collect*Async; AllSql closure; FROM/JOIN whitelist; dialect), PgTargetAnomalyTests (all re-pinned above), PgTargetSharedSwitchRoutingTests (delegation equality — the anomaly arm delegates), PgTargetThresholdLineageTests (every new comparison carries a marker or only trivial literals), PgTargetMcpSurfaceTests, StoreSqlClockDisciplineTests, FactSourceRegistryTests, LiveCleanupConversionRatchetTests + LivePostgresCollectionHygieneTests (the new live class is [Collection("live-postgres")] with LiveStoreCleanup.RunAsync), AuroraOnlySqlIsGatedTests (no new file mentions aurora_stat_), DocCommentHygiene (green in-worktree), RepoFileAdoptionTests (the new test reads no repo files).

Which component(s) does this affect?

• Lite
• Darling
• Lite Tests
• Darling Tests
• SQL collection scripts
• Documentation
• Full Dashboard (deprecated)
• CLI Installer (deprecated)

How was this tested?

Above. No production store or fleet server was read; the calibration numbers quoted are the coordinator's announced pass (measurements-3691 §A4 / §A7).

Checklist

• I have read the contributing guide
• My code builds with zero warnings (dotnet build -c Debug)
• I have tested my changes against at least one SQL Server version (the SQL Server pass is unreachable by every new arm; the routing pins that proved v1's byte-identity pass unchanged)
• I have not introduced any hardcoded credentials or server names

Partial for #3691 (lane 15). CHANGELOG deliberately untouched — the coordinator splices this wave's entries. Not armed (review-dark window).