Repository navigation
PostgreSQL-target analysis engine: v2/v3 slices and the v1 lanes' residue (from #3542) #3691
Description
Status: a work queue, not a parking lot. v1 of the PostgreSQL-target analysis engine landed on 2026-09-19 (#3542 closed with the exit check's evidence — nine lanes, twelve PRs incl. #3690). This issue holds the plan's v2 / v3 slices and the residue the v1 lanes reported at path:line. Nothing here waits on a ruling except the two marked maintainer decision; everything else is a lane. The validated plan (V1-BUILD-PLAN.md) and the design doc are on the design bus; the lane protocol that built v1 (one family per lane, file-disjoint partials, threshold_lineage on every bar, D1–D10) carries forward unchanged.
First: the calibration pass (unblocks honest bars everywhere)
- Every PostgreSQL bar shipped tonight is
threshold_lineage = 0(unmeasured). → measured 2026-09-19 10:24Z over 50 Aurora PostgreSQL clusters (comment below); the lineage-flip lane and the Aurora-suppression lane follow from it. The first v2 read is the fleet-distribution pass the SQL Server side got in The scorer's wait thresholds carry their measurement: WRITELOG stops saturating on a third of routine windows, deadlocks grade 5/hr apart from 90/hr, and the absolute gates scale with the hours actually observed (#3538 A5/A7) #3616: per-server distributions over the dogfood PostgreSQL targets for top-1 statement share (pg_statement_stats), temp bytes/sec and TPS (pg_database_stats), peak sessions / ceiling (pg_session_states), wait fraction per type (pg_wait_stats, Aurora), checkpoint requested/timed ratio (pg_write_stats), autovacuum backlog ratio (pg_autovacuum). Then re-derive each bar the V120 way (WARNING near the top of the routine mode, CRITICAL inside a measured empty interval) and flip the lineage comments to measured. Announce-before-firing on the fleet-watch seat; sanitized results on this issue.
v2 slices (from the plan §6)
- IO latency facts + anomaly (PostgreSQL I/O family: an operator now learns how slow each data-file read was, on measured Aurora-storage bars, and hears why when the pass cannot know — timing off or a pre-16 major (partial #3691) #3726) (trackedness-aware:
track_io_timingoff → unavailable, never 0.000 ms — the MCP payload-contract campaign: the agent-facing API tells the truth (~30 defects + an 11-rule contract) #3541 A8 lesson onget_pg_io_stats). - Replication lag + slot retention stories (§3.5) (Replication lag and slot retention stories for the PostgreSQL-target engine: a standby's stage and drift, a slot's disk-fill and horizon, on the alert's own bars (partial #3691) #3727) — constants shared with the alert evaluator (D9).
-
PG_BLOAT_TREND(§3.4) (PG_BLOAT_TREND: a PostgreSQL operator learns which table is actually accumulating space over fourteen days — growth of the estimate behind a size floor, never the spot percentage that reads 86% on a 2 MB table (partial #3691) #3722) offpg_table_bloat/pg_index_bloatestimates (a NULLest_bloat_pctis "suppressed", never zero). -
PG_IDLE_IN_TRANSACTION(§3.10) (PostgreSQL sessions family: an operator now learns which application left a transaction open and walked away — PG_IDLE_IN_TRANSACTION on measured duration bars, a band higher when it pins the xmin horizon, joined to the saturation story only when parked sessions fill the pool (partial #3691, lane 14) #3733) — the xmin holder kind lane 4 names; the saturation ↔ idle-in-transaction edge lane 3 left as a commented hook. - Active-query facts; blocking facts +
BlockingChainReconstructorport (PostgreSQL blocking family: an operator now learns who headed a blocking chain, in what state, and how long the sessions behind it had really waited — PG_BLOCKING_CHAIN from the sampled edges, PG_LOCK_WAIT_EVENTS from the engine's own log lines, PG_LONG_RUNNING_QUERY from the active sessions, and a blocked-sessions anomaly whose baseline counts a quiet capture as zero (partial #3691, lane 17) #3750) — PG blocking is SAMPLED sightings; Ingest target PostgreSQL logs as telemetry — errors, spills, autovacuum runs, and connection churn are invisible between counter samples #3601'slock_waitlog events are the event-grain source (also the Brains-review campaign: deferred structural residue (from #3538 / #3539 / #3540 / #3541) #3653 PG blocking-band item). - Deadlock exemplar drill-down (PostgreSQL deadlock cards carry the captured exemplars, grouped by shape and ranked by recurrence, so an operator learns the access pattern and not just the count (#3691) #3721) (
pg_deadlocksgraphs) beside the counter-derivedPG_DEADLOCK_RATE(which already says how many the log captured vs the engine counted). -
PG_WAL_VOLUME_SHIFT(§3.11) (The write family tells Aurora the truth: checkpoint pressure and max_wal_size are not applicable where storage owns checkpointing, and WAL volume is graded against its own baseline where the engine reports it (partial #3691) #3728) — the key exists and the.Write.csedge into checkpoint pressure is declared; the fact is not emitted. - (A PostgreSQL operator now learns which database is growing, how fast, and when it doubles: PG_DATABASE_GROWTH trends pg_database_size_stats over fourteen days, never a spot size, counts the databases the role cannot size instead of reading them as zero, and the instance's growth rate gets its own anomaly — every bar unmeasured and saying so (partial #3691) #3831 —
PG_DATABASE_GROWTH+ANOMALY_PG_DATABASE_GROWTHoverpg_database_size_stats, trend never spot, all bars unmeasured) Object growth; contention anomalies DONE asANOMALY_PG_BLOCKING(PostgreSQL blocking family: an operator now learns who headed a blocking chain, in what state, and how long the sessions behind it had really waited — PG_BLOCKING_CHAIN from the sampled edges, PG_LOCK_WAIT_EVENTS from the engine's own log lines, PG_LONG_RUNNING_QUERY from the active sessions, and a blocked-sessions anomaly whose baseline counts a quiet capture as zero (partial #3691, lane 17) #3750).PG_MAINTENANCE_SHAPE_SHIFT(§3.2) → v3: measured autovacuum worker concurrency is 0–1 on every cluster, ≥ 3 never (calibration B6) — no signal to baseline on this fleet. - (PostgreSQL targets learn whether the memory configuration fits the host: the §4b worst-case sum against the window's smallest memory_total_bytes roots an advisory at 0.4 and reaches the incident line only when host-memory pressure or temp spills co-fire, the host's sustained reclaimable share is graded, and a stock target reads unavailable (partial #3691, lane 32) #3809 —
CONFIG_PG_MEMORY_OVERCOMMIT+PG_HOST_MEMORY_PRESSURE;max_wal_sizevs disk stays out: disk free not collected) Composition checks (§4b):shared_buffers + max_connections × work_memvs host RAM;max_wal_sizevs disk — blocked on the host-RAM collector below. - Offered-vs-delivered co-fire — re-expressed as the anomaly pair (session spike fired, TPS anomaly not) on the calibration's evidence; the raw TPS trend was noise (§A1) (PostgreSQL sessions family: an operator now learns when a pool near its ceiling is accepting more connections than this hour usually carries while completing no more work — the offered-vs-delivered co-fire reads the anomaly pair, and the raw TPS-trend hook is retired on the calibration's evidence (partial #3691, lane 18) #3747) — lane 3's commented hook in
PgTargetScorer.Sessions.csreadsMetadata["trend"]; lane 6 shippedPgTargetScorer.TpsTrendKey = "tps_trend"— rename + uncomment (two lines) once the TPS trend is judged stable.
v3 slices (from the plan §6)
-
Viewer / worker surface for
collection_caveats(today only the MCP payloads and one Warning log line carry them — A pass whose fact families could not be read says so: analyze_server and get_analysis_facts carry collection_caveats on both SKUs, so a blind pass stops reading as an all-clear (partial #3691) #3763). — SPLIT 2026-09-22: the WORKER surface is DONE — A scheduled analysis pass's failed family reads are now readable on get_collection_health instead of dying in a log line (#3691) #3881 (lane 45: a process-wideCollectionCaveatLedgerevery analysis pass writes, scheduled or not;get_collection_healthon both SKUs carriesanalysis_caveatswith per-family persistence when a recent pass could not read a family; theanalyze_nowresult says the same); the VIEWER surface is WAITING-ON a Windows build seat — the Viewer is WPF, the coordinator's Mac cannot build or execute it, and CI would be its first execution; brief on request if CI-first verification is acceptable. — DONE 2026-09-24: the Viewer surface shipped in Persist analysis collection caveats in a store table (V141, #3691 part a1) #4154 (rung V141,collect.analysis_collection_caveats) and 3691 part a2 — Viewer shows persisted analysis collection caveats #4155 (a section on the SQL Server Collection Health tab and the PostgreSQL Overview tab, rendered on a Windows VM; comment 5813059565). -
A string seam on
Fact(or aDrillDown) so top-N cards can name more than their worst object (Fact.Metadatais doubles-only — A PostgreSQL table that turned autovacuum off and fell behind is its own card: CONFIG_PG_AUTOVACUUM_DISABLED names the table, how far past its own trigger line it sits and for how long, and joins the backlog's story on the same table (#3691 step 22) #3761, PG_BLOAT_TREND: a PostgreSQL operator learns which table is actually accumulating space over fourteen days — growth of the estimate behind a size floor, never the spot percentage that reads 86% on a 2 MB table (partial #3691) #3722, The vacuum family of a PostgreSQL analysis pass: a table persistently past its own autovacuum trigger line is named with its ratio, slope and whether autovacuum ran and lost, the wraparound counter and the xmin holder grade on the very constants the Tier-0 alerts page on, and the three chain into one story (#3542 lane 4) #3674 all hit it). — A summary card names the objects it ranked instead of counting them: three PostgreSQL cards stop hiding table names in metadata keys (#3691) #3864 (lane 43; typedFact.Rankedlist capped at 3,[0]pinned to the fact's own object;rankedon both engines'analyze_serverroot_fact andget_analysis_factscards only when ≥ 2 ride; the autovacuum-disabled rank keys, the bloatgrowth_*_<name>prefixes and the one-row backlog card retired; SQL Server and every non-ranking card byte-identical, pinned) -
The greedy single-path walk orphans config leaves not reachable from the path's tail (
CONFIG_PG_MAINT_WORK_MEMroots alone at 0.6 beside a backlog → wraparound → hold chain) — engine item, both engines. — A configuration lever hanging off a diagnostic chain is attached to that finding instead of orphaned into a card of its own, so a vacuum backlog and the memory bound that caps how fast it can clear arrive as one story (#3691) #3850 (lane 42;AnalysisStory.SideLeafKeys/side_leaveson both engines, classified by SOURCE becauseCONFIG_PG_MAINT_WORK_MEMis evidence-gated and not an advisory root; path/hash untouched; SQL Server structurally cannot produce one — pinned) -
(A PostgreSQL operator now learns which statement got slower when its plan changed, and why: PG_PLAN_REGRESSION joins the captured plan_hash flip to the statement's per-call step, PG_PARAMETER_SENSITIVITY names the skewed predicate column behind a multi-plan statement, and the server-wide per-call mean is baselined (partial #3691, lane 27) #3798) Plan regression (statement step-change ×
plan_hashflip viapg_statement_stats+ captured plans); parameter sensitivity (plan_hashvariance ×top_value_frequencyskew). -
(A PostgreSQL operator now learns which large relation a statement reads sequentially under a selective predicate, how often, and what an index would cost — PG_SEQ_SCAN_ADVISORY from plan_json Seq Scan nodes beside pg_qualstats selectivity and pg_table_bloat_stats size, evidence first (partial #3691, lane 30) #3811 — the PostgreSQL card, built under the refined discipline) Index-candidate analysis — RULED 2026-09-20 (refined): the analysis starts from the measured-slow query; a candidate/request is corroboration, never the driver; impact labelled "one operator's estimated cost" (never a % or a promise), uses labelled plan-cache-bounded, sample counts labelled sampled; every suggestion card states that a new index can cause regressions elsewhere and must be tested; advisory band; never a defining characteristic of the engine. Applies to BOTH SKUs. Lane 30 builds the PostgreSQL card under it.
pg_qualstats/ plan-JSON candidates surface as predicate, evaluation count, sample rate, selectivity, estimate error; noCREATE INDEXtext anywhere (the standing no-missing-index-folklore rule). -
(A PostgreSQL operator now learns which large relation a statement reads sequentially under a selective predicate, how often, and what an index would cost — PG_SEQ_SCAN_ADVISORY from plan_json Seq Scan nodes beside pg_qualstats selectivity and pg_table_bloat_stats size, evidence first (partial #3691, lane 30) #3811, evidence-first with the regression caveat pinned);plan_jsonSeq-Scan advisorieskernel-stats CPU decomposition (waiting vs burning)(A PostgreSQL operator learns how many cores their statements are burning and whether the backends are burning or waiting: pg_stat_kcache cores busy as the self-hosted CPU proxy graded only against the server's own hour-of-week routine, a burning-versus-waiting decomposition that becomes a finding on its own instrument's co-fire, and one honest unavailable fact on the measured fleet where the extension does not exist (#3691 lane 28) #3800);buffer-composition drill-down ((PostgreSQL buffer-pressure cards carry what the cache HOLDS — the latest pg_buffer_usage capture as pool fill, the ten largest resident relations, the index/heap/TOAST split and the cold share, folded into the advice, so an operator learns whether shared_buffers is one relation or a working set (#3691) #3795 + The buffer-composition drill-down's pressure block carries the fact's counts and observed span rather than a second copy of its per-second quotients, and its rounding helper reads whole — the two censuses #3795's pre-CI merge reddened on dev (#3691) #3799 + The member scan's known-truncated roster names the buffer drill-down's nullable rounder (#3691 lane 29 residue; dev red since #3795) #3804); Aurora peakmem per-query metadata; extension/meta advisories beyond the v1 pair; self-hosted CPU proxy. — remainder WAITING-ON measurement (2026-09-22): Aurora peakmem per-query needspg_buffer_usage)aurora_stat_*columns whose presence on the fleet is unmeasured (a D-batch read first); extension/meta advisories beyond the v1 pair have no failure population behind them (the two v1 checks came from measured gaps); the self-hosted CPU proxy has population 0 (pg_stat_kcacheabsent fleet-wide, batch C §C6/D6). None is idle-by-choice: each is a bar with no evidence to set it against yet.
Collector follow-ups feeding later slices (each trips ReadmeDerivedCountPinTests by design)
- (PostgreSQL targets get a per-database size series (pg_database_size_stats, hourly, a year) and the host's memory as six byte columns on pg_cpu_utilization's Performance Insights row (Darling V136), so the object-growth and memory-composition lanes have a source (partial #3691) #3794, V136 — six memory columns on
pg_cpu_utilizationfrom Performance Insights; stock: no row) Host RAM for PostgreSQL targets — the composition checks' blocker and the cross-server-transfer blocker the review named. - Disk free space;
per-database size series(PostgreSQL targets get a per-database size series (pg_database_size_stats, hourly, a year) and the host's memory as six byte columns on pg_cpu_utilization's Performance Insights row (Darling V136), so the object-growth and memory-composition lanes have a source (partial #3691) #3794,pg_database_size_stats, hourly, 400 d); per-database settings. — lane 46 (2026-09-22): per-database settings (pg_db_role_setting) as a snapshot besidepg_server_config;disk free spaceSTRUCK 2026-09-22: no in-engine SQL source on either engine (stock PostgreSQL has no free-bytes function —pg_database_size/pg_tablespace_sizeare USED bytes, already collected; Aurora exposesFreeLocalStorageonly through CloudWatch, and the collectors speak SQL only); the store host's own disk is already watched by the service's disk-pressure self-alert. Per-database settings DONE — A pg_server_config row now says whose setting it is, so a database that overrides work_mem stops being invisible (#3691) #3882 (lane 46; rung V138:database_name/role_nameonpg_server_config, the collector UNIONspg_db_role_setting, all ten server-wide reads exclude the overrides and one selects them — pinned by anchor — andget_pg_server_configpublishesdatabase_overrides). -
numbackendsonpg_database_stats(pg_database_stats stores numbackends and pg_wait_sampling stores sampled_ms (Darling V133), so a PostgreSQL operator's saturation fraction has a numerator and the sampler arm's wait rate has its duty-cycle denominator (partial #3691) #3716, V133 — stored; consumer switch PostgreSQL sessions family: an operator's connection-saturation card now divides pg_stat_database.numbackends — the universal one-minute level, summed over databases at one instant, peaked over the window — by the ceiling, and names the exception-capture peak as its stated fallback; both readings ride the fact, the bars do not move (partial #3691, lane 25) #3782 — the saturation numerator isnumbackends, capture peak as the stated fallback) — a RUNG (universal 1-min client-only numerator for saturation instead of exception-capture peaks). Ladder-sequenced with the other seat.
Residue the v1 lanes reported (path:line; each S unless marked)
-
Lineage flip, round 3 (calibration D):
ANOMALY_PG_BAD_ACTOR_SHARE's reuse ofPgRatioAnomalyThreshold 3.0→ measured for statement share (≈ p99.1 of per-statement hour-of-week ratios, p99.6 with the busy/share floor; window fenced at 2026-09-19 16:40Z, before the .453 regression); the keyed per-queryid mean-ms ratio → measured (3.0 ≈ p99.3). Memory / growth bars stay unmeasured until V136 is installed on the fleet (D1/D2/D4 re-fire then). S. — ANOMALY_PG_PLAN_REGRESSION reads each flipped statement's OWN hour-of-week per-call mean; the server-wide mean is the cold fallback — the third calibration read's lineage lands (partial #3691) #3845 (lane 39; server-wide mean kept as the cold fallback,seriesstamp 1/0; no bar value changed) -
Switch
ANOMALY_PG_PLAN_REGRESSIONto the keyed per-queryid mean-ms series (lane 27's follow-up, now EVIDENCED by D5: the server-wide mean's hour-of-week ratio never reaches 3.0 across 50 clusters in a week (max 4.4, p99 2.05 — a 90× per-statement step dilutes to nothing), the keyed series has a real tail (p99.9 57×)). The seam exists (A baseline series can now be scoped to ONE statement: the per-key seam on PgBaselineProvider (a keyed GetBaselineAsync overload, a third resolve seam, $7 after the Q6 clock) and two per-queryid arms — a statement's share of the collection's execution time and its per-call mean — so a PostgreSQL bad actor can be judged against its OWN hour-of-week habit; no consumer yet (partial #3691, lane 33) #3810); the detector partialPgTargetAnomalyDetector.Plans.csreads the keyed arm for each candidate statement and folds ontoPG_PLAN_REGRESSIONfor the SAME queryid. S/M. — ANOMALY_PG_PLAN_REGRESSION reads each flipped statement's OWN hour-of-week per-call mean; the server-wide mean is the cold fallback — the third calibration read's lineage lands (partial #3691) #3845 (lane 39; server-wide mean kept as the cold fallback,seriesstamp 1/0; no bar value changed) -
(The PostgreSQL engine absorbs what waves 4–5 reported back: get_pg_database_stats surfaces numbackends, get_pg_cpu_utilization carries the host's memory, the spill leads to the memory composition check, ANOMALY_PG_CPU_BURN is the load family's stock second confirmer, autovacuum_enabled = off reads as off, the replication e2e is anchored, and four stale docs say what is true now (partial #3691, between waves) #3832 item 8 —
NOT option_value::boolean, all seven spellings verified live) BUGPgAutovacuumStatsCollector.cs:143/:161lower(option_value) = 'false'— PostgreSQL stores reloptions as typed (off/0/no/f), soautovacuum_disablednever reads true andCONFIG_PG_AUTOVACUUM_DISABLEDcannot fire (lane 36, verified live) → lane 37 item 8. -
PG_XMIN_HOLDreads 0 when several equally-old transactions alternate asis_winner— the identity arm sees each win once (PgTargetFactCollector.Vacuum.cs:~220-247; lane 36 saw a 4.3 M-xid horizon score 0). Design: count the HORIZON's persistence, attribute the winner separately. M. — PG_XMIN_HOLD measures the HORIZON's persistence and attributes the holder separately — five equally old transactions taking turns no longer score 0 on a horizon they held for 25 minutes (partial #3691) #3844 (lane 40; the horizon's persistence is graded on the longest run above the bar across captures, attribution is a separate arm) -
An all-zero baseline history yields
EffectiveStdDev = 0→ the absolute fallback (BaselineBucket.cs:~98):ANOMALY_PG_BLOCKINGsat at 0.5 with 92 blocked sessions against a clean month. A zero-history bucket should be the STRONGEST evidence, not none — shared engine, both SKUs. M. — A month of zeros is the strongest baseline there is, not the absence of one: a zero-history bucket now fires as an extremity instead of routing to the young-store fallback bar (#3691) #3849 (lane 41;BaselineBucket.IsZeroHistory+ the extremity arm at the top ofAnomalyGate.Decide, both SKUs; 23 pre-existing calibration pins untouched; the planted SQL Server store has no such bucket so its output is unchanged) -
D8 — acceptance read for the zero-history arm (A month of zeros is the strongest baseline there is, not the absence of one: a zero-history bucket now fires as an extremity instead of routing to the young-store fallback bar (#3691) #3849) on the keyed share series: count
ANOMALY_PG_BAD_ACTOR_SHAREfires carryingbaseline_zero_history = 1and their share values in the first week of .458 (D3B measured 21% of top-5-statement hour-of-week cells as zero-history). If the rate is dominated by appearances just over the 0.05 share floor, raise that family's zero-history magnitude floor to a measured bar. WAITING-ON install + one week: fire ≥ 2026-09-29 03:30Z (pgmon,analysis_findings/facts read, announce first). — DONE 2026-09-24: 71 of 1,990 share findings were zero-history (3.6 %, 16 clusters), median window share 8.6 %; at most 34 % could be near the floor, so the floor is unchanged (comment above). -
(The PostgreSQL engine absorbs what waves 4–5 reported back: get_pg_database_stats surfaces numbackends, get_pg_cpu_utilization carries the host's memory, the spill leads to the memory composition check, ANOMALY_PG_CPU_BURN is the load family's stock second confirmer, autovacuum_enabled = off reads as off, the replication e2e is anchored, and four stale docs say what is true now (partial #3691, between waves) #3832 items 2+3)
PgTargetToolRecommendationsnamesget_pg_cpu_utilizationfor the memory keys but that tool does not project the six memory columns (PostgreSQL targets get a per-database size series (pg_database_size_stats, hourly, a year) and the host's memory as six byte columns on pg_cpu_utilization's Performance Insights row (Darling V136), so the object-growth and memory-composition lanes have a source (partial #3691) #3794's positive census names the collector as the only reader) — extendDarlingPgCpuUtilizationReader(+ census) or re-point the row atget_analysis_facts source=pg_memory; andPG_TEMP_SPILL → CONFIG_PG_MEMORY_OVERCOMMITedge fromPgTargetRelationshipGraph.Query.cs(source-owns-edges; the amplifier already lifts). S, next shared batch (PostgreSQL targets learn whether the memory configuration fits the host: the §4b worst-case sum against the window's smallest memory_total_bytes roots an advisory at 0.4 and reaches the incident line only when host-memory pressure or temp spills co-fire, the host's sustained reclaimable share is graded, and a stock target reads unavailable (partial #3691, lane 32) #3809). -
(The PostgreSQL engine absorbs what waves 4–5 reported back: get_pg_database_stats surfaces numbackends, get_pg_cpu_utilization carries the host's memory, the spill leads to the memory composition check, ANOMALY_PG_CPU_BURN is the load family's stock second confirmer, autovacuum_enabled = off reads as off, the replication e2e is anchored, and four stale docs say what is true now (partial #3691, between waves) #3832 item 7 —
ANOMALY_PG_CPU_BURNis a load-family confirmer; lane 36's measured gap closed) Wire the stock second confirmer (A PostgreSQL operator learns how many cores their statements are burning and whether the backends are burning or waiting: pg_stat_kcache cores busy as the self-hosted CPU proxy graded only against the server's own hour-of-week routine, a burning-versus-waiting decomposition that becomes a finding on its own instrument's co-fire, and one honest unavailable fact on the measured fleet where the extension does not exist (#3691 lane 28) #3800 out-of-lane):PgTargetScorer.Anomaly.csLoadAnomalyAmplifiers—ANOMALY_PG_CPU_BURNfired joins the load-family confirmers andAnomalyAmplifiers(AnomalyCpuBurn)gets the load arms (today the burn anomaly is a solo card under the 1.49 cap);AnomalyToFamilies[AnomalyCpuBurn]folds onto base-0PG_CPU_BURN_CORES(unreachable) — considerCpuDecompositionas the family. Gate on lane 36's measurement. S, shared file. -
(The PostgreSQL engine absorbs what waves 4–5 reported back: get_pg_database_stats surfaces numbackends, get_pg_cpu_utilization carries the host's memory, the spill leads to the memory composition check, ANOMALY_PG_CPU_BURN is the load family's stock second confirmer, autovacuum_enabled = off reads as off, the replication e2e is anchored, and four stale docs say what is true now (partial #3691, between waves) #3832 item 1)
get_pg_database_stats(DarlingPgDatabaseReader.cs:~20) does not surfacenumbackends— a level-only saturation card has no tool that shows the count behind it; add the column to that read (+ README :572 / runbook sentence from pg_database_stats stores numbackends and pg_wait_sampling stores sampled_ms (Darling V133), so a PostgreSQL operator's saturation fraction has a numerator and the sampler arm's wait rate has its duty-cycle denominator (partial #3691) #3716). S. -
(The PostgreSQL engine absorbs what waves 4–5 reported back: get_pg_database_stats surfaces numbackends, get_pg_cpu_utilization carries the host's memory, the spill leads to the memory composition check, ANOMALY_PG_CPU_BURN is the load family's stock second confirmer, autovacuum_enabled = off reads as off, the replication e2e is anchored, and four stale docs say what is true now (partial #3691, between waves) #3832 item 5 — every planted timestamp anchored on the test's
as_of)PgTargetReplicationTests.ADriftingStandbyAndAnInactiveGrowingSlot…is a live FLAKE (red on two PR runs — lanes 24 and 26 — green on dev's re-run each time): timing assumption in the fixture; pin it deterministic. S. -
(PostgreSQL sessions: a window with no session capture still grades connection saturation from pg_stat_database.numbackends as a level-only card that says its states are unobserved, and the Aurora wait-profile advice names the window mean against its routine (partial #3691, lane 26) #3791 — level-only card; the redacted-majority-WITH-captures half remains a ruling) A window with NO session capture still emits no saturation fact although
numbackendscan now supply the numerator alone — the level-only card is the next honest step (PostgreSQL sessions family: an operator's connection-saturation card now divides pg_stat_database.numbackends — the universal one-minute level, summed over databases at one instant, peaked over the window — by the ceiling, and names the exception-capture peak as its stated fallback; both readings ride the fact, the bars do not move (partial #3691, lane 25) #3782 out-of-lane); under a redacted-majority window the ratio is still withheld thoughpg_stat_databaseis unprivileged — ruling. S. -
(A baseline series can now be scoped to ONE statement: the per-key seam on PgBaselineProvider (a keyed GetBaselineAsync overload, a third resolve seam, $7 after the Q6 clock) and two per-queryid arms — a statement's share of the collection's execution time and its per-call mean — so a PostgreSQL bad actor can be judged against its OWN hour-of-week habit; no consumer yet (partial #3691, lane 33) #3810 seam + A PostgreSQL bad actor is now graded against its OWN normal: ANOMALY_PG_BAD_ACTOR_SHARE judges each top statement's window share against that statement's hour-of-week share bucket and walks into its PG_BAD_ACTOR card, which becomes a context band under the story line with the absolute share bars as named measured-routine context, so a statement at its usual 60% is the workload and a statement at 55% that usually takes 10% is the story (partial #3691, lane 34) #3827 re-grade:
ANOMALY_PG_BAD_ACTOR_SHAREagainst the statement's own hour-of-week share, absolute share as context) RULED 2026-09-20 → lanes 33 (per-key baseline seam) + 34: re-grade the bad-actor share as deviation from the statement's OWN hour-of-week share baseline, absolute share as context (derive-not-constant; the percentile raise rejected). (was: bad-actor share bars 0.25 / 0.60 are ROUTINE given the busy floor (85 % / 44 % of busy hours on 24 clusters) — raise to ≈ p90 / p99 of busy hours (0.85 / 0.90), or re-grade as deviation from the statement's own hour-of-week share baseline with the absolute share as context. Maintainer decision; then a lane. -
(PostgreSQL wait standouts, the I/O admission floor and the ratio multiple say what the second fleet read found: measured per family, cited 2026-09-20, and every Aurora wait fact stops calling a validated bar a guess (#3691 lineage flip, round 2) #3785) Lineage flip, round 2 (calibration C):
PgRatioAnomalyThreshold 3.0→ measured for TPS (≈ p99.97) with the per-family note that for waits the ratio alone is routine (p93) and the magnitude floor (8× fleet p99) is the gate; wait standouts 0.15 / 1.0 → measured (p99.6 / p99.99 of non-IO event buckets);IoBaselineBucketMinimumReads 250→ measured (57 % of fleet 15-min buckets excluded; hour-of-day tier for half the fleet stated); sessions bars keep thenumbackendsnote (lane 25); sampled-wait bars stay unmeasured (population 0). S. -
(PostgreSQL sessions: a window with no session capture still grades connection saturation from pg_stat_database.numbackends as a level-only card that says its states are unobserved, and the Aurora wait-profile advice names the window mean against its routine (partial #3691, lane 26) #3791)
PgTargetAdvice.Anomaly.csComposeWaitProfileRatio(~:258) does not name the window mean the Aurora fact now carries (mean_ms_per_sec/mean_modified_z); the sampled composer (PgTargetAdvice.Wait.cs:~205) does — The wait-profile detector judged the window peak alone: its inline gate becomes the shared AnomalyGate pair call every other baseline detector took in #3724, so one hot collection in a quiet window stops reading as a profile shift (#3741) #3773's advice enrichment has no PG-target twin. S. -
(Stock PostgreSQL's sampled wait profile fires its first-occurrence reading on the peak's bar alone, like its Aurora and SQL Server twins — a young server's one hot sampler cycle is no longer held by a mean it has no baseline to judge (partial #3691, lane 35, ruled 2026-09-20) #3808) RULED 2026-09-20 → lane 35: the sampled twin matches the unsampled twin's
is_newgating (peak-only), lineage marked unmeasured, revisit if apg_wait_samplingcluster ever exists. (was: the sampled twin (PgTargetAnomalyDetector.WaitsSampled.cs:~204) pair-gates itsis_newarm; The wait-profile detector still fires on the window peak alone: its inline gate has no mean, unlike every detector #3724 pair-gated (Q1 residue) #3741/The wait-profile detector judged the window peak alone: its inline gate becomes the shared AnomalyGate pair call every other baseline detector took in #3724, so one hot collection in a quiet window stops reading as a profile shift (#3741) #3773 ruled peak-only for the SQL Server and Aurora arms — pick one for the PG family. Maintainer decision. -
Aurora wait-profile detector lacks the peak-AND-mean gate (The PostgreSQL Long-Running Query read applied excludedDatabases AFTER LIMIT, so an excluded database's sessions consumed the page and the alert came back short or empty while matches existed; the list is now the candidates CTE's third flag ahead of the cap, counted on the card (Closes #3742) — and the Aurora wait-profile detector gains the peak-AND-mean gate its twins have (partial #3691) #3780) its sampled twin has (The anomaly gate judges the window peak AND the window mean, so one hot sample stops reading as an anomaly and a longer window stops buying more of them; I/O reads the pair like its siblings (Partial #3653 — item 1, Q1) #3724's gate; lane 9 root body
PgTargetAnomalyDetector.cs) — parity item, one arm. S. -
PostgreSQL-target buckets stay UTC-keyed after Q6 (Hour-of-week baselines key on the target's local clock, not UTC: one shared step-function expression re-keys every arm through the scaffold, three bound clock parameters from server_properties (zone id, offset fallback, UTC when there is no row), the lookup through the same numbers — nothing keyed is stored, so re-bucketing is the next compute (#3653 item 12, Q6) #3749) (PostgreSQL targets key their hour-of-week buckets on their own TimeZone setting, so a New York target's Tue 09:00 bucket is 09:00 in New York in every season — the clock read becomes the provider's second seam, UTC becomes the fallback instead of the rule (partial #3691, lane 20) #3757): the local-clock re-keying reads
server_propertiesfor the zone, and PostgreSQL targets have no such row — a PG-target clock source is needed (pg_settings.TimeZonefrompg_server_configis the likely one; the collector reads it already) feedingBaselineLocalClockforPgTargetBaselineProvider. Until then every PG hour-of-week baseline is UTC hour-of-week (stated, not hidden). M. -
PgTargetBaselineProvider.csroot doc says the base binds only$2/$3(PostgreSQL targets key their hour-of-week buckets on their own TimeZone setting, so a New York target's Tue 09:00 bucket is 09:00 in New York in every season — the clock read becomes the provider's second seam, UTC becomes the fallback instead of the rule (partial #3691, lane 20) #3757, incl. theTpsTrendKeyprose) — since Hour-of-week baselines key on the target's local clock, not UTC: one shared step-function expression re-keys every arm through the scaffold, three bound clock parameters from server_properties (zone id, offset fallback, UTC when there is no row), the lookup through the same numbers — nothing keyed is stored, so re-bucketing is the next compute (#3653 item 12, Q6) #3749 the shared bind path also supplies$4..$6(transition instant + offsets); one paragraph. S — next shared-file batch, with lane 18's falsifiedTpsTrendKeyprose (PgTargetScorer.Database.cs:50–51,PgTargetFactCollector.Database.cs:23–24, 129–131, 261–265). -
compare_analysisbands the v2 baselined metrics absolutely (compare_analysis sigma-bands the v2 PostgreSQL baselined metrics — I/O read latency, replay lag, WAL volume — so a 16× latency move stops reading by the ladder alone; the write family has one definition of "WAL tracked" (partial #3691) #3748) (v2 exit check):ComparisonBanding.BaselinedMetricFormaps the v1 four only;PG_IO_READ_LATENCY_MS,PG_REPLICATION_LAG,PG_WAL_VOLUME_SHIFTcome backband_source: absolutewhile their detectors say 25σ. + one predicate for "WAL tracked" inPgTargetFactCollector.Write.cs. (lane 19) -
A CRITICAL leaf consumed mid-chain is never NAMED by the root (The root card names the leaves it consumed: a story's root advice now says who was behind the chain — the parked application, the lead blocker, the job — so a CRITICAL leaf walked under a higher root is no longer nameless (both engines, partial #3691) #3759 — story text on both engines; a typed
named_hopspayload array is a follow-up if wanted: ephemeral list onAnalysisFinding+ the two story→finding maps) (v2 exit check, both engines):PG_IDLE_IN_TRANSACTION1.2 naming its holder sits underPG_CONNECTION_SATURATION1.25; the root's advice andco_firednever carry the holder's identity — an operator reading only the card cannot find the application.InferenceEngine.BuildStories/ story advice composition: the root's advice should carry named leaves' identifiers. Design pass (M, shared, both SKUs). -
Aurora-suppress
PG_CHECKPOINT_PRESSUREandCONFIG_PG_MAX_WAL_SIZE(The write family tells Aurora the truth: checkpoint pressure and max_wal_size are not applicable where storage owns checkpointing, and WAL volume is graded against its own baseline where the engine reports it (partial #3691) #3728) (calibration A4): Aurora reports requested-share 0 with 60 synthetic timed "checkpoints" per hour on every cluster and does not populatepg_stat_wal; onaurora-postgresthe fact is a permanent measured 0 that means nothing and the convention check is noise. Emitnot_applicablewith the reason in metadata; the checkpoint → max_wal_size story must not form there. -
Lineage flip to measured (PostgreSQL bars the fleet measured now say so: lineage comments cite the 2026-09-19 read and threshold_lineage flips to 1, so a validated bar stops calling itself a guess (#3691 lineage flip) #3723) (calibration A/B): TPS floor 50 / fallback 500, temp 1 / 10 MiB/s, backlog 3 / 10 / 5, deadlock 5 / 20, bad-actor busy floor 0.05 (+ share bars conditional on it), wait 0.20 / 1.0 / io 0.40 / 2.0, CPU 80 / 95 — comments cite the date and population,
threshold_lineage0 → 1 on those facts; sessions 0.8 / 0.9 stay engine-defined with the measured note (fleet max 10.3 % of ceiling). -
compare_analysisbands a 6× TPS move "stable" (compare_analysis sigma-bands the PostgreSQL baselined metrics, so a 6× TPS surge on a PostgreSQL target stops reading "stable" (#3691 W1/W2) #3713) (exit check W1) —ComparisonBanding.BaselinedMetricFor(ComparisonBanding.cs:~159-166) maps onlyCPU_SQL_PERCENT / CPU_SPIKE / IO_READ_LATENCY_MS / SESSION_STATS; lane 9'sMetricNames.PgTps(same unit, same difference) is unmapped andPG_TPSbase severity is 0 so the absolute ladder can never move it; the same window's detector calls it 25σ. MapPG_TPS → PgTps,PG_CONNECTION_SATURATION → PgSessionCount,PG_DEADLOCK_RATE → PgDeadlockRate,PG_CPU_PERCENT → PgCpu. Alsoplan_cache_churn.noteprose saysBAD_ACTOR_<hash>in a PG payload (W2, harmless). -
ANOMALY_PG_WAIT_PROFILEstays solo and can never escape the cap (v2 plumbing for the PostgreSQL-target engine: the I/O, replication and bloat families exist as stubs, and the wait-profile anomaly folds onto its wait and can page (partial #3691) #3715) —AnomalyIncidentReconciler.ResolveFamiliesresolves the dominant contributor only for the literalANOMALY_WAIT_PROFILE;PgTargetFactKeys.AnomalyToFamilieshas no wait entry;FactScorer.IsExtremeAnomalyhas no PG wait-profile arm. Shared-file item. -
PG baseline sources are unfloored at purge (PostgreSQL baselines survive a shortened retention, and pg_cpu carries the CPU dispersion floor, so a 7-day PG retention no longer silently switches anomaly detection off (#3691) #3711) —
DarlingRetention.BaselineServingRawCollectorsfloors onlycpu_utilization/file_io_stats;pg_database_stats,pg_session_states,pg_wait_stats,pg_cpu_utilizationare user-editable and a shortened retention silently starves the PG baselines (PgBaselineProvider times out client-side scanning raw history, and tiered retention will mask it rather than fix it #1757's shape). Defaults ≥ 30 d are pinned; the runtime floor is the missing half. -
BaselineMath.AbsStdDevFloorForhas no arm forMetricNames.PgCpu(PostgreSQL baselines survive a shortened retention, and pg_cpu carries the CPU dispersion floor, so a 7-day PG retention no longer silently switches anomaly detection off (#3691) #3711) (SQL CPU carries 5.0). -
get_analysis_factsnever runs the anomaly detector on EITHER engine (get_analysis_facts runs the anomaly detector on both engines, so the facts read stops omitting the detector's ANOMALY_* facts (#3691) #3714) (DarlingAnalysisService.CollectAndScoreFactsAsync= collector + scorer; Lite twin) — anomaly lineage/gate metadata is visible only in findings. Pre-existing; twin edit. -
(rung pg_database_stats stores numbackends and pg_wait_sampling stores sampled_ms (Darling V133), so a PostgreSQL operator's saturation fraction has a numerator and the sampler arm's wait rate has its duty-cycle denominator (partial #3691) #3716 / V133; consumer read + sampled anomaly arm Stock PostgreSQL's sampled waits read per second the sampler was watching (V133 sampled_ms — ten times what the interval arithmetic said on the service sampler), get their own baseline and ANOMALY_PG_SAMPLED_WAIT_PROFILE detector folding onto the dominant sampled wait, and the exact source wins when both wait tables write one window (partial #3691, lane 24) #3765) Stock PostgreSQL's sampled waits get no anomaly baseline (lane 9, by design) and the Service-side wait sampler for stock PostgreSQL — targets without pg_wait_sampling get no wait accumulation at all #3604 service-sampler arm stores
profile_period_ms = 1000but samples 30 s of each 300 s cycle, soΔsamples × period / observedis a ~10× floor on that arm — the instrument token lives incollector_state, outside the analysis FROM/JOIN whitelist. Whitelistcollector_statefor analysis reads, or storesampled_msper row (rung). "Both wait sources non-empty" is not detected (read-then-fallback reads one). -
Saturation ↔ Lock-wait mesh in
PgTargetRelationshipGraph.Saturation.csis safe only while saturation has no other leaf — revisited as the leaves landed: idle-in-transaction (PostgreSQL sessions family: an operator now learns which application left a transaction open and walked away — PG_IDLE_IN_TRANSACTION on measured duration bars, a band higher when it pins the xmin horizon, joined to the saturation story only when parked sessions fill the pool (partial #3691, lane 14) #3733, share-gated), Lock → idle (The PostgreSQL engine absorbs what the v2 lanes reported back: the I/O and WAL baseline tables are floored, the I/O baseline restores hour-of-week at a quarter-hour grain, the WAL co-fire reads the anomaly and not a base-0 fact, a Lock wait reaches its idle holder, and the blocking family exists as stubs for lane 17 (partial #3691, between waves) #3737), Lock/idle → blocking chain (PostgreSQL blocking family: an operator now learns who headed a blocking chain, in what state, and how long the sessions behind it had really waited — PG_BLOCKING_CHAIN from the sampled edges, PG_LOCK_WAIT_EVENTS from the engine's own log lines, PG_LONG_RUNNING_QUERY from the active sessions, and a blocked-sessions anomaly whose baseline counts a quiet capture as zero (partial #3691, lane 17) #3750); saturation-alone byte-identity pinned each time. -
Per-table
CONFIG_PG_AUTOVACUUM_DISABLED(design §3.1) has no key (A PostgreSQL table that turned autovacuum off and fell behind is its own card: CONFIG_PG_AUTOVACUUM_DISABLED names the table, how far past its own trigger line it sits and for how long, and joins the backlog's story on the same table (#3691 step 22) #3761); v1 folds the reloption into the backlog fact.wal_level/archive_modeposture facts (§4c did not list them). -
audit_configfor a PostgreSQL target answers the honestnot_collectedenvelope (plumbing); aCONFIG_PG_*projection into the same tool is a description-contract change — decide. — DECIDED yes (Erik, 2026-09-22) → audit_config answers a PostgreSQL target with its own settings: the CONFIG_PG_* facts projected into the same recommendations shape, engine in place of edition, and the two checkpointing knobs marked not applicable on Aurora (#3691 line 70) #3867 (lane 44; theCONFIG_PG_*facts project into the SQL Server envelope withengineforedition, stringcurrent_values with units, nosuggested_value, a fourth statusnot_applicablefor Aurora's checkpoint knobs; the description changed once and the three edition-aware doc comments say resource-based) -
PG_MONITORING_PERMISSIONSat 0.5 — RULED 2026-09-20: 0.5 stands (measured blindness outranks the 0.4 advisory band); no code change. -
(A pass whose fact families could not be read says so: analyze_server and get_analysis_facts carry collection_caveats on both SKUs, so a blind pass stops reading as an all-clear (partial #3691) #3763 + The collection-caveat live pin counts missing families, not reads: step 22's second pg_autovacuum_stats read is the same one missing vacuum family (follow-up to #3763, partial #3691) #3767, both SKUs; Viewer surface for the caveats is a v3 candidate) Per-family collector failures are logged, not carried on the context: a pass where every family silently failed reads as a normal pass (both SKUs, pre-existing);
emptyprose cannot distinguish "facts scored, nothing fired" from "no fact emitted" without a fact-count surface. -
PgSampledWaitProfileFallbackMsPerSecdeliberately undeclared (Stock PostgreSQL's sampled waits read per second the sampler was watching (V133 sampled_ms — ten times what the interval arithmetic said on the service sampler), get their own baseline and ANOMALY_PG_SAMPLED_WAIT_PROFILE detector folding onto the dominant sampled wait, and the exact source wins when both wait tables write one window (partial #3691, lane 24) #3765 declared it with its reader) — arrives with the sampled anomaly arm. -
MEASURED 2026-09-20 (comment): NOT a gap in general — four stock stories cross 1.5 (two with zero baseline); the one capped family is the load-anomaly pair, fixed by wiring
ANOMALY_PG_CPU_BURNas a confirmer (lane 37 item 7). RULED 2026-09-20 → lane 36 MEASURES it: a stock PostgreSQL in local Docker under a pgbench load storm, the real collectors writing a throwaway store,analyze_servercounting reachable families and the max achievable severity — the measurement decides whether the ceiling is a gap. (was: Notification on a STOCK target under a load storm cannot cross 1.5 on anomalies alone (the second confirmer isPG_CPU_PERCENT ≥ 80, Aurora-only) — the self-hosted CPU proxy (v3) is what closes it; until then it is stated, not hidden.
Not here
- Brains-review campaign: deferred structural residue (from #3538 / #3539 / #3540 / #3541) #3653 — the brains-review structural residue (SQL Server side + shared).
- analyze_server for PostgreSQL targets — the inference engine answers 'what is wrong, ranked' for one engine only #3605 — folded into v1 as a design input (its three contributions landed in the plumbing).
Activity
erikdarlingdata commentedon Sep 19, 2026
Campaign started 09:55Z (on the maintainer's go; v1 is live on the fleet as of this morning's install, schema 131).
Wave 1 — five lanes in flight, file-disjoint, all from dev
f52bf531:pg_io/pg_replication/pg_bloatsources, the IO / replication / slot / bloat / WAL-anomaly keys,MetricNamesconstants), stub partials with markers for lanes 11–13, the shared-switch arms, the FROM/JOIN whitelist for the v2 tables — and the one v1 behavioural residue that lives in shared files:ANOMALY_PG_WAIT_PROFILEgets its family resolution and its extremity-escape arm.compare_analysissigma-bands the PostgreSQL baselined metrics (W1) and the plan-cache note stops speaking SQL Server in a PostgreSQL payload (W2).AbsStdDevFloorForgets itspg_cpuarm.get_analysis_factsruns the resolved anomaly detector on BOTH engines (twin edit).numbackendsonpg_database_stats,sampled_msonpg_wait_sampling; collectors write, nothing reads yet (consumer lanes follow once the columns have data).Wave 2 (content, after P2 merges): lane 11 IO latency + anomaly, 12 replication lag + slot retention, 13 bloat trend, 14 idle-in-transaction (+ the saturation hook), 15 WAL volume shift, 16 deadlock exemplar drill-down.
Coordinator's own item: the fleet calibration pass (announced production reads; sanitized distributions posted here) — designed while wave 1 builds.
Checklist ticks land as each PR merges; partials reference
#3691and quote their line.erikdarlingdata commentedon Sep 19, 2026
#3691 fleet calibration — the measured population (2026-09-19, 10:24–10:29Z)
Population: 50 Aurora PostgreSQL clusters (majors 16–17), the whole PostgreSQL side of the dogfood fleet. Every shape below is an Aurora shape. Stock-PostgreSQL bars stay
unmeasuredunless the quantity is engine-neutral by construction (TPS, temp bytes/s, sessions/ceiling, backlog ratio, deadlocks/h, statement share, IO ms/op, bloat growth, idle-in-transaction duration); wait-type fractions, CPU/ACU, checkpoint/WAL are Aurora-specific (and two of them are Aurora-absent — §A4/§A7).Windows: 14 d for 1-minute counters (pre-bucketed to 5-min rates), 7 d for
pg_session_statescaptures, statements andpg_wait_stats. Reads were store-side aggregates (server_idonly), EXPLAIN-verified chunk-pruned; 11 + 13 statements, ~2 min total wall. Hypertable spans:pg_database_statsfrom 08-25,pg_write_statsfrom 09-08,pg_cpu_utilizationfrom 09-01,pg_log_eventsfrom 09-19 (today)."Per-server pNN" = the distribution of that server's own hourly / 5-min values; "across servers" = the fleet spread of those per-server statistics.
A — v1 bars
A1
PG_TPS/ANOMALY_PG_TPS(AnomalyThresholds.PgTpsFloor 50, PgTpsFallback 500, PgRatioAnomalyThreshold 3.0)Per-server p50 TPS: 6.5 – 79 (median 31). Per-server p99: 6.8 – 205 (median 59). Per-server 5-min max: median 1,710 (≈ 50× its own p50) — every cluster has a routine-time burst of that size somewhere in 14 d, which is exactly why the detector is hour-of-week baselined and not absolute.
stats_resetbuckets: 0 in 14 d.Verdict: floor 50 ≈ the fleet's p75 of per-server p50 — a median cluster's anomalies are gated out at its routine load and admitted at spikes; fallback 500 is above every cluster's p99 and below its routine bursts. Keep; lineage → measured.
A2
PG_TEMP_SPILL(TempSpillConcerningBytesPerSec 1 MiB/s, TempSpillCriticalBytesPerSec 10 MiB/s)Per-server p50 temp bytes/s: 0 – 1.06 MB (median 46 KB/s); p99: median 58 KB/s, fleet p90 313 KB/s, max 2.06 MB/s. Share of 5-min buckets ≥ 1 MiB/s: median 0, fleet p90 0.27 %, one cluster 61.8 % (a chronic spiller — the finding should be permanent there, and it is correct). Share ≥ 10 MiB/s: max 0.42 %.
Verdict: WARNING 1 MiB/s ≈ p99.7 of fleet buckets; CRITICAL 10 MiB/s inside the measured empty interval for 49/50. Both bars validated → measured.
A3
PG_CONNECTION_SATURATION(0.8 / 0.9,IdleInTransactionShareBar 0.25)Usable ceilings 1,240 – 5,000; 50/50 have a
max_connectionssnapshot. Sessions/ceiling per capture: per-server p99 median 0.021, fleet max 0.103. Idle-in-transaction share: per-server max median 6.3 %, fleet max 16.7 %; ≥ 0.25 never.Verdict: engine-defined bars stand; measured note: the fleet never exceeded 10.3 % of its ceiling in 7 d — the bar is deep in the empty interval, as it should be for a hard-FATAL cliff. Caveat:
total_sessionshere is the exception-capture peak;numbackends(V133, R4) is the honest numerator and re-measurement follows it.A4
PG_CHECKPOINT_PRESSURE(CheckpointRequestedShareConcerning 0.5 / Critical 0.9, MinimumCheckpoints 3) — NOT APPLICABLE ON AURORAAll 50: requested share p99 = 0, max 0.033; 60 "checkpoints" per hour on every cluster (one per minute, timed) — Aurora's checkpointer counters are synthetic; Aurora storage owns checkpointing and
max_wal_sizehas no effect on it.Verdict: on
aurora-postgresthe fact is a permanent measured 0 that means nothing, andCONFIG_PG_MAX_WAL_SIZE's convention check is noise there. → residue lane: Aurora-suppressPG_CHECKPOINT_PRESSURE(emitnot_applicablewith the reason in metadata) and theCONFIG_PG_MAX_WAL_SIZEadvisory; the checkpoint → max_wal_size story must not form on Aurora. Stock bars stay unmeasured.A5
PG_AUTOVACUUM_BACKLOG(BacklogPersistenceSamples 3 (hourly), BacklogCriticalRatio 10, BacklogManyTables 5)Hourly max backlog ratio per server: p50 median 0.92 (tables live just under their own line — autovacuum fires at 1.0, so this is the engine working), p99 median 1.0, fleet p90 of p99 2.2, one table at 350×. Hours with ratio ≥ 1: median 2.1 % of hours, max 21.5 %. Longest consecutive run ≥ 1: median 1 h, fleet p90 3 h, max 8 h. Hours ≥ 10×: max 0.6 %. Tables past their line at once: max 3; ≥ 5 never.
Verdict: persistence 3 consecutive hourly samples ≈ the fleet p90 of longest runs (WARNING shape, top of routine); critical ratio 10 ≈ p99.4 (empty interval); many-tables 5 above the fleet max. All three → measured.
A6
PG_DEADLOCK_RATE(DeadlockWarnPerHour 5, Critical 20; anomaly floor 1/h)40/50 clusters: zero deadlocks in 14 d. The other 10: 1 – 4 total, max 2 in any hour. Bars 5/h and 20/h never approached.
Verdict: bars in the measured empty interval; the first-occurrence anomaly (≥ 1/h) is the one that fires here (0.3 – 0.9 % of hours on the 10). → measured.
A7
PG_WAL_VOLUME_SHIFT(metadata only in v1; lane 15's target) — NOT MEASURABLE ON AURORApg_write_stats.wal_bytes= 0 on all 50 for the full span (Aurora does not populatepg_stat_wal).Verdict: lane 15 must emit
unavailable("WAL volume not reported by this engine") on Aurora and build the shift only for engines that report it; no measured shape exists.A8
PG_BAD_ACTOR_*(BadActorShareConcerning 0.25 / Critical 0.60, BadActorBusyFloor 0.05)Top-1 statement share of hourly exec time, per server: p50 median 0.32 (range 0.14 – 0.67), p90 median 0.42, p99 median 0.71, max up to 0.95. Hours with share ≥ 0.25: median 63 % of hours; ≥ 0.60: median 4 %, fleet p90 29 %. Busy fraction (Σ exec time ÷ wall): p50 median 0.7 %, p90 median 3.3 %, max 69 %; hours below the 0.05 busy floor: median 100 %, fleet p25 73 %, min 12 %.
Verdict: the share bars alone are inside routine (a 63 %-of-hours shape) — an idle cluster's one busy statement is not a bad actor; the busy floor is the discriminator and it is well placed (≈ fleet p97 of hourly busy fractions). Keep 0.05 → measured; keep 0.25/0.60 as measured-conditional ("given busy ≥ 0.05") — a second read conditioned on the floor is the follow-up if the fired rate on the fleet looks wrong.
B — v2 bars
B1 IO read latency (lane 11) —
pg_io_statsΔread_time_ms/Δreads, hourlytrack_io_timingeffective on 50/50 (no all-zero timing). Per-server p50 ms/read: 1.0 – 10 (median 1.29); p99: median 7.3, fleet p90 26, max 33.7; reads/hour p50 spans 1 – 790 k (a p99 over 4 reads is noise — the lane needs a reads-per-hour floor).Shape for the lane: WARNING ≈ 10 ms (≈ fleet p90 of per-server p99), CRITICAL ≈ 30 ms (≈ fleet max), both conditioned on a reads floor (≥ 1,000 reads/hour excludes the noisy tail);
unavailablewhenread_time_msis all-zero (timing off). Engine-neutral quantity → measured (Aurora storage); state that stock local-disk latency is a different population.B2 Replication (lane 12) — thin
One cluster has a standby:
replay_bytes_behindsteady 886 – 892 KB (a steadily N behind shape, not drifting);replay_lag_msNULL on every row (Aurora reports bytes, not lag time). One logical slot, inactive,wal_status = reserved, retained 1.3 – 1.76 MB.Shape for the lane: the fact must tolerate NULL
*_lag_ms(bytes are the primary; the design already says so); the slot bar (SlotRetainedWalWarningBytes10 GB, shared with the alert) is far above the one data point. Lineage stays unmeasured (n = 1).B3 Bloat (lane 13) — spot % is noise here, exactly as the design predicted
Table
bloat_pct_estimatewhere the estimate is available: per-server p50 median 12.9 % but fleet p75 71 %, p90 86 %, max 99.8 % (small heaps — p50 heap 75 MB — inflate the estimate). 14-day growth per table: p50 0 bytes on most servers; p99 median 5.6 MB / 6.3 %; fleet p90 874 MB / 120 %; max 6.8 GB / 523 %. Indexest_bloat_pct(where not skipped): p50 46 %, p90 69 %, p99 88 %, max 100 % — with median index size 16 KB; most indexes skipped (3 – 4 skip reasons per server).Shape for the lane: trend only — growth ≥ 25 % AND ≥ 256 MB over the window ≈ the fleet p90 of worst-table growth; a heap-size floor (≥ 64 MB) before any % is spoken; index bloat needs an index-size floor (≥ 64 MB) or it is 100 % noise. → measured.
B4 Idle in transaction (lane 14)
Per-server
xact_duration_msof idle-in-transaction rows: p50 24 ms, p99 50 ms, max 27.7 s on one cluster; rows ≥ 1 min: 0 across 50 clusters × 7 d;state_is_redactedshare 0.Shape for the lane: a 60 s WARNING / 10 min CRITICAL duration floor is inside the measured empty interval (the chronic-holder shape is absent on this fleet, which is the point); the redaction guard has no exercise here. → measured (empty interval).
B5 Wait fractions (lane 5's bars:
WaitRollupConcerning 0.20 / Critical 1.0, WaitIoConcerning 0.40 / Critical 2.0)Fleet 5-min buckets, ms waited per second observed:
iop50 0.012, p90 0.077, p99 0.46, max 3.9 (≥ 0.20: 2.9 % of buckets; ≥ 1.0: 0.27 %);ipcp99 0.35, max 7.8;lockp99 0.034, max 2.0;lwlock/lsn/bufferpinnegligible. Per-server total non-CPU: p50 median 0.012, p99 median 0.062, fleet p90 of p99 0.82, max 10.7; share ≥ 0.20 median 0, fleet p75 2.8 %, worst 41 % (a chronically waiting cluster); ≥ 1.0 fleet p90 0.5 %, worst 15.7 %.io≥ 0.40: fleet p90 2.4 %, worst 24 %.Verdict: 0.20 ≈ fleet p97 of buckets, 1.0 ≈ p99.7, io 0.40 ≈ p99, io 2.0 ≈ p99.9. All four → measured (Aurora
aurora_stat_system_waits).B6 Autovacuum worker concurrency (
PG_MAINTENANCE_SHAPE_SHIFT, design §3.2)Per capture: mean 0.05 workers, p99 = 1, max 2, ≥ 3 never; flat across hour-of-day.
Verdict: no signal on this fleet (default
autovacuum_max_workers3, concurrency 0 – 1) → deprioritise to v3; noted on the issue.B7 Deadlock exemplars (lane 16)
7 clusters with
pg_deadlocksrows: 1 – 4 each, distinct graphs, all 2-participant, every one with a victim statement. (approximate_row_countsaid 0 — the unanalysed-table trap; chunks carry the truth.)B8 CPU as percent of configured ACU capacity (
CpuCapacityWarningPercent 80 / Critical 95)Per-server p50: median 6.3 % of ceiling; p99 median 49 %, fleet p90 88 %; max 100 % on ≥ 75 % of clusters (Serverless reaches its ceiling briefly). Share ≥ 80: median 0.2 %, max 3.6 %; ≥ 95: median 0.1 %, max 1.2 %. Raw
cpu_percentp99 = 100 on ≥ 10 % of clusters while the ACU capacity in use idles at 0.5 – 4.5 of 8 – 128 configured — the raw number saturates at tiny capacity, confirming "never raw cpu_percent".Verdict: 80 ≈ p99.8, 95 ≈ p99.9 → measured (Aurora Serverless v2).
Consequences filed on #3691
PG_CHECKPOINT_PRESSURE+CONFIG_PG_MAX_WAL_SIZE(A4) — new residue lane.unavailableon Aurora (A7).PG_MAINTENANCE_SHAPE_SHIFT→ v3 (B6).threshold_lineagemetadata 0 → 1 on those facts; sessions 0.8/0.9 stay engine-defined with the measured note.PostgreSQL baseline sources survive a shortened retention, and pg_cpu…
compare_analysis sigma-bands the PostgreSQL baselined metrics, so a 6…
v2 plumbing for the PostgreSQL-target engine: vocabulary, stubs, swit…
get_analysis_facts runs the resolved anomaly detector on both engines…
pg_database_stats stores numbackends and pg_wait_sampling stores samp…
PostgreSQL bars the fleet measured now say so: lineage comments and t…
PostgreSQL deadlock cards carry the captured exemplars, grouped by sh…
PG_BLOAT_TREND / PG_INDEX_BLOAT_TREND: the bloat family grades fourte…
PostgreSQL bars the fleet measured now say so: lineage comments cite …
PostgreSQL I/O family: read latency per operation from pg_stat_io, gr…
RUNG-GATED: the Long-Running Query opt-out knob's Darling store home …
Replication lag and slot retention stories for the PostgreSQL-target …
92 remaining items
erikdarlingdata commentedon Sep 24, 2026
PR #4154 merged to dev as 688d43c (2026-09-24 10:30Z). This is part a1: store rung V141 adds
collect.analysis_collection_caveats. The analysis pass now writes its collection caveats to it on a best-effort basis, so they outlive the service process, and the daily retention sweep prunes it. The Viewer section that reads it follows in a separate PR. This issue stays open.erikdarlingdata commentedon Sep 24, 2026
PR #4155 merged to dev as 6b8311c (2026-09-24 11:16Z). This is part a2: the Darling Viewer now reads
collect.analysis_collection_caveats(V141, #4154). It lists the data families the scheduled analysis pass currently can't read, with the reason and the first and last time each was seen. The list is on the Collection Health tab for SQL Server servers and on the Overview tab for PostgreSQL targets. This issue stays open for D8 and the items waiting on measurements.erikdarlingdata commentedon Sep 24, 2026
The viewer surface for
collection_caveatshas shipped.collect.analysis_collection_caveats. The analysis pass writes the current caveats after each run, best-effort, and the daily purge prunes rows older than 7 days.Together these finish the viewer half of this item. The worker half shipped earlier in #3881.
Still open here:
erikdarlingdata commentedon Sep 24, 2026
D8 read (the #3849 zero-history arm on the keyed statement-share series). This is the PostgreSQL store's .458 window: 2026-09-22 03:13:06Z to the install restart at 2026-09-24 22:04:50Z, about 2.8 days. It's a read-only query of
collect.analysis_findings.ANOMALY_PG_BAD_ACTOR_SHAREfindings on 46 of the 50 clusters. 71 are zero-history fires (3.6 %), on 16 clusters. They're identified by the zero-history story shape ("against its own routine of 0 % ± 0 % for this hour-of-week", with the display-capped σ).PgTargetScorer.BadActorShareConcerning, while the fact's stored value is the window SHARE (≤ the peak). That's why 16 values sit under 5 %. So at most 24 of 71 (34 %) could be peaks just over the floor (the 16 under 5 % plus the 8 in 5–6 %). Even that upper bound is not "dominated".Nothing changes in the new build for this family: the keyed share series is excluded from #3653 B's per-hour tiles, and #4180's young-bucket walk never overrides a zero-history bucket. So this read also describes the post-install behaviour.
erikdarlingdata commentedon Sep 25, 2026
Aurora peak-memory presence read (the line-28 remainder)
The remainder said that Aurora peakmem per-query "needs
aurora_stat_*columns whose presence on the fleet is unmeasured (a D-batch read first)". This read measures that presence from the collected store. It ran no query on an Aurora target.aurora_stat_statements(true)intopg_statement_statsand storesmax_exec_peakmem_bytes. The read called the Darling MCP toolget_pg_top_querieson each cluster of the PostgreSQL store. Each call asked for the last 24 hours and the top 50 statement shapes by total execution time. It ran on 2026-09-25, between 00:40Z and 00:50Z.max_exec_peakmem_bytesis not null for every one of the top 50 shapes: 2,500 of 2,500 rows.Verdict
The prerequisite is now measured. The column is present and populated on the whole fleet. This item now waits on a design decision, not on a measurement:
max_exec_peakmem_bytesto the per-query PostgreSQL findings and advisories. On a target that is not Aurora, the value is null, not zero, like the other Aurora-only fields.The other two remainders do not change. The extension and meta advisories beyond the v1 pair still have no failure population. The self-hosted CPU proxy still has a population of 0, because
pg_stat_kcacheis absent on 240 of 240 (server, database) pairs (batch D, D6).Merge origin/dev into fix/3953-qs-interval-latest, renumber V140 to V142
Match the peak-memory sentence's actual FormatBytes rounding in its o…
Fix a swapped column-index read in PgTargetTopStatementsSql (#3691)
Attach Aurora per-query peak memory as a context fact (#3691) (#4370)
erikdarlingdata commentedon Sep 26, 2026
Claude posting for Erik Darling
Aurora per-query peak memory now appears as context on PostgreSQL per-query findings (no threshold, no grading effect; absent off Aurora): #4370, merged to dev (9861eb8). The other #3691 items stay open.
erikdarlingdata commentedon Sep 26, 2026
Claude posting for Erik Darling
Closing: everything on this issue has shipped, except two items that wait on data or time. Those are split into their own issues.
Shipped
CONFIG_PG_MEMORY_OVERCOMMITre-grade at peak observed backends: CONFIG_PG_MEMORY_OVERCOMMIT tells a PostgreSQL operator whether the backends that actually showed up could overrun the box, with what the settings permit stated beside it (#3691 lane 47) #3907.collection_caveatsviewer surface: Persist analysis collection caveats in a store table (V141, #3691 part a1) #4154 (store rung V141) and 3691 part a2 — Viewer shows persisted analysis collection caveats #4155.Split out
Not carried forward
pg_stat_kcacheis absent on 240 of 240 (server, database) pairs (batch D, D6). If that extension appears on a measured target, this becomes its own issue.Release 3.9.0 (#4936)
Show the store's collection caveats on the web Collection Health tabs (…