Skip to content

PostgreSQL-target analysis engine: v2/v3 slices and the v1 lanes' residue (from #3542) #3691

Closed

Description

@erikdarlingdata

Status: a work queue, not a parking lot. v1 of the PostgreSQL-target analysis engine landed on 2026-09-19 (#3542 closed with the exit check's evidence — nine lanes, twelve PRs incl. #3690). This issue holds the plan's v2 / v3 slices and the residue the v1 lanes reported at path:line. Nothing here waits on a ruling except the two marked maintainer decision; everything else is a lane. The validated plan (V1-BUILD-PLAN.md) and the design doc are on the design bus; the lane protocol that built v1 (one family per lane, file-disjoint partials, threshold_lineage on every bar, D1–D10) carries forward unchanged.

First: the calibration pass (unblocks honest bars everywhere)

v2 slices (from the plan §6)

v3 slices (from the plan §6)

Collector follow-ups feeding later slices (each trips ReadmeDerivedCountPinTests by design)

Residue the v1 lanes reported (path:line; each S unless marked)

Not here

Activity

  1. erikdarlingdata commented on Sep 19, 2026

    @erikdarlingdata
    OwnerAuthor

    Campaign started 09:55Z (on the maintainer's go; v1 is live on the fleet as of this morning's install, schema 131).

    Wave 1 — five lanes in flight, file-disjoint, all from dev f52bf531:

    • P2 plumbing — the v2 vocabulary declared once (pg_io / pg_replication / pg_bloat sources, the IO / replication / slot / bloat / WAL-anomaly keys, MetricNames constants), stub partials with markers for lanes 11–13, the shared-switch arms, the FROM/JOIN whitelist for the v2 tables — and the one v1 behavioural residue that lives in shared files: ANOMALY_PG_WAIT_PROFILE gets its family resolution and its extremity-escape arm.
    • R1 — compare_analysis sigma-bands the PostgreSQL baselined metrics (W1) and the plan-cache note stops speaking SQL Server in a PostgreSQL payload (W2).
    • R2 — the four PG baseline sources join the retention floor (PgBaselineProvider times out client-side scanning raw history, and tiered retention will mask it rather than fix it #1757's shape); AbsStdDevFloorFor gets its pg_cpu arm.
    • R3 — get_analysis_facts runs the resolved anomaly detector on BOTH engines (twin edit).
    • R4 — RUNG V133 — numbackends on pg_database_stats, sampled_ms on pg_wait_sampling; collectors write, nothing reads yet (consumer lanes follow once the columns have data).

    Wave 2 (content, after P2 merges): lane 11 IO latency + anomaly, 12 replication lag + slot retention, 13 bloat trend, 14 idle-in-transaction (+ the saturation hook), 15 WAL volume shift, 16 deadlock exemplar drill-down.

    Coordinator's own item: the fleet calibration pass (announced production reads; sanitized distributions posted here) — designed while wave 1 builds.

    Checklist ticks land as each PR merges; partials reference #3691 and quote their line.

  2. erikdarlingdata commented on Sep 19, 2026

    @erikdarlingdata
    OwnerAuthor

    #3691 fleet calibration — the measured population (2026-09-19, 10:24–10:29Z)

    Population: 50 Aurora PostgreSQL clusters (majors 16–17), the whole PostgreSQL side of the dogfood fleet. Every shape below is an Aurora shape. Stock-PostgreSQL bars stay unmeasured unless the quantity is engine-neutral by construction (TPS, temp bytes/s, sessions/ceiling, backlog ratio, deadlocks/h, statement share, IO ms/op, bloat growth, idle-in-transaction duration); wait-type fractions, CPU/ACU, checkpoint/WAL are Aurora-specific (and two of them are Aurora-absent — §A4/§A7).
    Windows: 14 d for 1-minute counters (pre-bucketed to 5-min rates), 7 d for pg_session_states captures, statements and pg_wait_stats. Reads were store-side aggregates (server_id only), EXPLAIN-verified chunk-pruned; 11 + 13 statements, ~2 min total wall. Hypertable spans: pg_database_stats from 08-25, pg_write_stats from 09-08, pg_cpu_utilization from 09-01, pg_log_events from 09-19 (today).

    "Per-server pNN" = the distribution of that server's own hourly / 5-min values; "across servers" = the fleet spread of those per-server statistics.

    A — v1 bars

    A1 PG_TPS / ANOMALY_PG_TPS (AnomalyThresholds.PgTpsFloor 50, PgTpsFallback 500, PgRatioAnomalyThreshold 3.0)

    Per-server p50 TPS: 6.5 – 79 (median 31). Per-server p99: 6.8 – 205 (median 59). Per-server 5-min max: median 1,710 (≈ 50× its own p50) — every cluster has a routine-time burst of that size somewhere in 14 d, which is exactly why the detector is hour-of-week baselined and not absolute. stats_reset buckets: 0 in 14 d.
    Verdict: floor 50 ≈ the fleet's p75 of per-server p50 — a median cluster's anomalies are gated out at its routine load and admitted at spikes; fallback 500 is above every cluster's p99 and below its routine bursts. Keep; lineage → measured.

    A2 PG_TEMP_SPILL (TempSpillConcerningBytesPerSec 1 MiB/s, TempSpillCriticalBytesPerSec 10 MiB/s)

    Per-server p50 temp bytes/s: 0 – 1.06 MB (median 46 KB/s); p99: median 58 KB/s, fleet p90 313 KB/s, max 2.06 MB/s. Share of 5-min buckets ≥ 1 MiB/s: median 0, fleet p90 0.27 %, one cluster 61.8 % (a chronic spiller — the finding should be permanent there, and it is correct). Share ≥ 10 MiB/s: max 0.42 %.
    Verdict: WARNING 1 MiB/s ≈ p99.7 of fleet buckets; CRITICAL 10 MiB/s inside the measured empty interval for 49/50. Both bars validated → measured.

    A3 PG_CONNECTION_SATURATION (0.8 / 0.9, IdleInTransactionShareBar 0.25)

    Usable ceilings 1,240 – 5,000; 50/50 have a max_connections snapshot. Sessions/ceiling per capture: per-server p99 median 0.021, fleet max 0.103. Idle-in-transaction share: per-server max median 6.3 %, fleet max 16.7 %; ≥ 0.25 never.
    Verdict: engine-defined bars stand; measured note: the fleet never exceeded 10.3 % of its ceiling in 7 d — the bar is deep in the empty interval, as it should be for a hard-FATAL cliff. Caveat: total_sessions here is the exception-capture peak; numbackends (V133, R4) is the honest numerator and re-measurement follows it.

    A4 PG_CHECKPOINT_PRESSURE (CheckpointRequestedShareConcerning 0.5 / Critical 0.9, MinimumCheckpoints 3) — NOT APPLICABLE ON AURORA

    All 50: requested share p99 = 0, max 0.033; 60 "checkpoints" per hour on every cluster (one per minute, timed) — Aurora's checkpointer counters are synthetic; Aurora storage owns checkpointing and max_wal_size has no effect on it.
    Verdict: on aurora-postgres the fact is a permanent measured 0 that means nothing, and CONFIG_PG_MAX_WAL_SIZE's convention check is noise there. → residue lane: Aurora-suppress PG_CHECKPOINT_PRESSURE (emit not_applicable with the reason in metadata) and the CONFIG_PG_MAX_WAL_SIZE advisory; the checkpoint → max_wal_size story must not form on Aurora. Stock bars stay unmeasured.

    A5 PG_AUTOVACUUM_BACKLOG (BacklogPersistenceSamples 3 (hourly), BacklogCriticalRatio 10, BacklogManyTables 5)

    Hourly max backlog ratio per server: p50 median 0.92 (tables live just under their own line — autovacuum fires at 1.0, so this is the engine working), p99 median 1.0, fleet p90 of p99 2.2, one table at 350×. Hours with ratio ≥ 1: median 2.1 % of hours, max 21.5 %. Longest consecutive run ≥ 1: median 1 h, fleet p90 3 h, max 8 h. Hours ≥ 10×: max 0.6 %. Tables past their line at once: max 3; ≥ 5 never.
    Verdict: persistence 3 consecutive hourly samples ≈ the fleet p90 of longest runs (WARNING shape, top of routine); critical ratio 10 ≈ p99.4 (empty interval); many-tables 5 above the fleet max. All three → measured.

    A6 PG_DEADLOCK_RATE (DeadlockWarnPerHour 5, Critical 20; anomaly floor 1/h)

    40/50 clusters: zero deadlocks in 14 d. The other 10: 1 – 4 total, max 2 in any hour. Bars 5/h and 20/h never approached.
    Verdict: bars in the measured empty interval; the first-occurrence anomaly (≥ 1/h) is the one that fires here (0.3 – 0.9 % of hours on the 10). → measured.

    A7 PG_WAL_VOLUME_SHIFT (metadata only in v1; lane 15's target) — NOT MEASURABLE ON AURORA

    pg_write_stats.wal_bytes = 0 on all 50 for the full span (Aurora does not populate pg_stat_wal).
    Verdict: lane 15 must emit unavailable ("WAL volume not reported by this engine") on Aurora and build the shift only for engines that report it; no measured shape exists.

    A8 PG_BAD_ACTOR_* (BadActorShareConcerning 0.25 / Critical 0.60, BadActorBusyFloor 0.05)

    Top-1 statement share of hourly exec time, per server: p50 median 0.32 (range 0.14 – 0.67), p90 median 0.42, p99 median 0.71, max up to 0.95. Hours with share ≥ 0.25: median 63 % of hours; ≥ 0.60: median 4 %, fleet p90 29 %. Busy fraction (Σ exec time ÷ wall): p50 median 0.7 %, p90 median 3.3 %, max 69 %; hours below the 0.05 busy floor: median 100 %, fleet p25 73 %, min 12 %.
    Verdict: the share bars alone are inside routine (a 63 %-of-hours shape) — an idle cluster's one busy statement is not a bad actor; the busy floor is the discriminator and it is well placed (≈ fleet p97 of hourly busy fractions). Keep 0.05 → measured; keep 0.25/0.60 as measured-conditional ("given busy ≥ 0.05") — a second read conditioned on the floor is the follow-up if the fired rate on the fleet looks wrong.

    B — v2 bars

    B1 IO read latency (lane 11) — pg_io_stats Δread_time_ms/Δreads, hourly

    track_io_timing effective on 50/50 (no all-zero timing). Per-server p50 ms/read: 1.0 – 10 (median 1.29); p99: median 7.3, fleet p90 26, max 33.7; reads/hour p50 spans 1 – 790 k (a p99 over 4 reads is noise — the lane needs a reads-per-hour floor).
    Shape for the lane: WARNING ≈ 10 ms (≈ fleet p90 of per-server p99), CRITICAL ≈ 30 ms (≈ fleet max), both conditioned on a reads floor (≥ 1,000 reads/hour excludes the noisy tail); unavailable when read_time_ms is all-zero (timing off). Engine-neutral quantity → measured (Aurora storage); state that stock local-disk latency is a different population.

    B2 Replication (lane 12) — thin

    One cluster has a standby: replay_bytes_behind steady 886 – 892 KB (a steadily N behind shape, not drifting); replay_lag_ms NULL on every row (Aurora reports bytes, not lag time). One logical slot, inactive, wal_status = reserved, retained 1.3 – 1.76 MB.
    Shape for the lane: the fact must tolerate NULL *_lag_ms (bytes are the primary; the design already says so); the slot bar (SlotRetainedWalWarningBytes 10 GB, shared with the alert) is far above the one data point. Lineage stays unmeasured (n = 1).

    B3 Bloat (lane 13) — spot % is noise here, exactly as the design predicted

    Table bloat_pct_estimate where the estimate is available: per-server p50 median 12.9 % but fleet p75 71 %, p90 86 %, max 99.8 % (small heaps — p50 heap 75 MB — inflate the estimate). 14-day growth per table: p50 0 bytes on most servers; p99 median 5.6 MB / 6.3 %; fleet p90 874 MB / 120 %; max 6.8 GB / 523 %. Index est_bloat_pct (where not skipped): p50 46 %, p90 69 %, p99 88 %, max 100 % — with median index size 16 KB; most indexes skipped (3 – 4 skip reasons per server).
    Shape for the lane: trend only — growth ≥ 25 % AND ≥ 256 MB over the window ≈ the fleet p90 of worst-table growth; a heap-size floor (≥ 64 MB) before any % is spoken; index bloat needs an index-size floor (≥ 64 MB) or it is 100 % noise. → measured.

    B4 Idle in transaction (lane 14)

    Per-server xact_duration_ms of idle-in-transaction rows: p50 24 ms, p99 50 ms, max 27.7 s on one cluster; rows ≥ 1 min: 0 across 50 clusters × 7 d; state_is_redacted share 0.
    Shape for the lane: a 60 s WARNING / 10 min CRITICAL duration floor is inside the measured empty interval (the chronic-holder shape is absent on this fleet, which is the point); the redaction guard has no exercise here. → measured (empty interval).

    B5 Wait fractions (lane 5's bars: WaitRollupConcerning 0.20 / Critical 1.0, WaitIoConcerning 0.40 / Critical 2.0)

    Fleet 5-min buckets, ms waited per second observed: io p50 0.012, p90 0.077, p99 0.46, max 3.9 (≥ 0.20: 2.9 % of buckets; ≥ 1.0: 0.27 %); ipc p99 0.35, max 7.8; lock p99 0.034, max 2.0; lwlock / lsn / bufferpin negligible. Per-server total non-CPU: p50 median 0.012, p99 median 0.062, fleet p90 of p99 0.82, max 10.7; share ≥ 0.20 median 0, fleet p75 2.8 %, worst 41 % (a chronically waiting cluster); ≥ 1.0 fleet p90 0.5 %, worst 15.7 %. io ≥ 0.40: fleet p90 2.4 %, worst 24 %.
    Verdict: 0.20 ≈ fleet p97 of buckets, 1.0 ≈ p99.7, io 0.40 ≈ p99, io 2.0 ≈ p99.9. All four → measured (Aurora aurora_stat_system_waits).

    B6 Autovacuum worker concurrency (PG_MAINTENANCE_SHAPE_SHIFT, design §3.2)

    Per capture: mean 0.05 workers, p99 = 1, max 2, ≥ 3 never; flat across hour-of-day.
    Verdict: no signal on this fleet (default autovacuum_max_workers 3, concurrency 0 – 1) → deprioritise to v3; noted on the issue.

    B7 Deadlock exemplars (lane 16)

    7 clusters with pg_deadlocks rows: 1 – 4 each, distinct graphs, all 2-participant, every one with a victim statement. (approximate_row_count said 0 — the unanalysed-table trap; chunks carry the truth.)

    B8 CPU as percent of configured ACU capacity (CpuCapacityWarningPercent 80 / Critical 95)

    Per-server p50: median 6.3 % of ceiling; p99 median 49 %, fleet p90 88 %; max 100 % on ≥ 75 % of clusters (Serverless reaches its ceiling briefly). Share ≥ 80: median 0.2 %, max 3.6 %; ≥ 95: median 0.1 %, max 1.2 %. Raw cpu_percent p99 = 100 on ≥ 10 % of clusters while the ACU capacity in use idles at 0.5 – 4.5 of 8 – 128 configured — the raw number saturates at tiny capacity, confirming "never raw cpu_percent".
    Verdict: 80 ≈ p99.8, 95 ≈ p99.9 → measured (Aurora Serverless v2).

    Consequences filed on #3691

    1. Aurora-suppress PG_CHECKPOINT_PRESSURE + CONFIG_PG_MAX_WAL_SIZE (A4) — new residue lane.
    2. Lane 15 (WAL shift): unavailable on Aurora (A7).
    3. Lane 11/13/14: bars above with floors; lane 12 stays unmeasured (n = 1).
    4. PG_MAINTENANCE_SHAPE_SHIFT → v3 (B6).
    5. Lineage flip lane: TPS floor/fallback, temp 1/10 MiB/s, backlog 3/10/5, deadlock 5/20, bad-actor busy floor 0.05 (+ conditional share), wait 0.20/1.0/0.40/2.0, CPU 80/95 → measured comments with this file's date and population; threshold_lineage metadata 0 → 1 on those facts; sessions 0.8/0.9 stay engine-defined with the measured note.
  3. 92 remaining items

  4. erikdarlingdata commented on Sep 24, 2026

    @erikdarlingdata
    OwnerAuthor

    PR #4154 merged to dev as 688d43c (2026-09-24 10:30Z). This is part a1: store rung V141 adds collect.analysis_collection_caveats. The analysis pass now writes its collection caveats to it on a best-effort basis, so they outlive the service process, and the daily retention sweep prunes it. The Viewer section that reads it follows in a separate PR. This issue stays open.

  5. erikdarlingdata commented on Sep 24, 2026

    @erikdarlingdata
    OwnerAuthor

    PR #4155 merged to dev as 6b8311c (2026-09-24 11:16Z). This is part a2: the Darling Viewer now reads collect.analysis_collection_caveats (V141, #4154). It lists the data families the scheduled analysis pass currently can't read, with the reason and the first and last time each was seen. The list is on the Collection Health tab for SQL Server servers and on the Overview tab for PostgreSQL targets. This issue stays open for D8 and the items waiting on measurements.

  6. erikdarlingdata commented on Sep 24, 2026

    @erikdarlingdata
    OwnerAuthor

    The viewer surface for collection_caveats has shipped.

    Together these finish the viewer half of this item. The worker half shipped earlier in #3881.

    Still open here:

    • D8, the zero-history acceptance read, which is dated on or after 2026-09-29, once the install has had a week;
    • the items waiting on measurement: Aurora peakmem per-query, extension and meta advisories, and the self-hosted CPU proxy. Each needs evidence it doesn't have yet.
  7. removed
    in-progressActively being worked by a local session or its agents (PR open or in flight)
    on Sep 24, 2026
  8. erikdarlingdata commented on Sep 24, 2026

    @erikdarlingdata
    OwnerAuthor

    D8 read (the #3849 zero-history arm on the keyed statement-share series). This is the PostgreSQL store's .458 window: 2026-09-22 03:13:06Z to the install restart at 2026-09-24 22:04:50Z, about 2.8 days. It's a read-only query of collect.analysis_findings.

    • Volume. 1,990 ANOMALY_PG_BAD_ACTOR_SHARE findings on 46 of the 50 clusters. 71 are zero-history fires (3.6 %), on 16 clusters. They're identified by the zero-history story shape ("against its own routine of 0 % ± 0 % for this hour-of-week", with the display-capped σ).
    • Their window shares: min 2.1 %, median 8.6 %, max 21.7 %. In bands: < 5 %: 16; 5–6 %: 8; 6–8 %: 11; 8–10 %: 11; 10–20 %: 24; ≥ 20 %: 1.
    • The floor. The zero-history arm fires on the window PEAK ≥ PgTargetScorer.BadActorShareConcerning, while the fact's stored value is the window SHARE (≤ the peak). That's why 16 values sit under 5 %. So at most 24 of 71 (34 %) could be peaks just over the floor (the 16 under 5 % plus the 8 in 5–6 %). Even that upper bound is not "dominated".
    • Ruling (per D8's own criterion): no change. The zero-history magnitude floor stays where it is. The arm's fires are a small minority (3.6 %) and mostly well above the floor.

    Nothing changes in the new build for this family: the keyed share series is excluded from #3653 B's per-hour tiles, and #4180's young-bucket walk never overrides a zero-history bucket. So this read also describes the post-install behaviour.

  9. erikdarlingdata commented on Sep 25, 2026

    @erikdarlingdata
    OwnerAuthor

    Aurora peak-memory presence read (the line-28 remainder)

    The remainder said that Aurora peakmem per-query "needs aurora_stat_* columns whose presence on the fleet is unmeasured (a D-batch read first)". This read measures that presence from the collected store. It ran no query on an Aurora target.

    • Source. The collector already reads aurora_stat_statements(true) into pg_statement_stats and stores max_exec_peakmem_bytes. The read called the Darling MCP tool get_pg_top_queries on each cluster of the PostgreSQL store. Each call asked for the last 24 hours and the top 50 statement shapes by total execution time. It ran on 2026-09-25, between 00:40Z and 00:50Z.
    • Result.
      • 50 of 50 clusters returned statement rows.
      • On all 50 clusters, max_exec_peakmem_bytes is not null for every one of the top 50 shapes: 2,500 of 2,500 rows.
      • The read had 0 errors.
      • Sample maxima: 7.3 MB on cluster A, 9.4 MB on cluster B and 79 MB on cluster C.
    • Limits. The read covers only the top 50 shapes of each cluster over 24 hours. It did not check the long tail. Each value is the high-water mark of one shape, not a total over a window.

    Verdict

    The prerequisite is now measured. The column is present and populated on the whole fleet. This item now waits on a design decision, not on a measurement:

    1. What "Aurora peakmem per-query metadata" means in the analysis engine. A proposed reading: attach max_exec_peakmem_bytes to the per-query PostgreSQL findings and advisories. On a target that is not Aurora, the value is null, not zero, like the other Aurora-only fields.
    2. If a finding needs a bar, measure the distribution first. Read the p50 and p99 of shape peak memory on each cluster. Compare them with the memory of the instance.

    The other two remainders do not change. The extension and meta advisories beyond the v1 pair still have no failure population. The self-hosted CPU proxy still has a population of 0, because pg_stat_kcache is absent on 240 of 240 (server, database) pairs (batch D, D6).

  10. erikdarlingdata commented on Sep 26, 2026

    @erikdarlingdata
    OwnerAuthor

    Claude posting for Erik Darling

    Aurora per-query peak memory now appears as context on PostgreSQL per-query findings (no threshold, no grading effect; absent off Aurora): #4370, merged to dev (9861eb8). The other #3691 items stay open.

  11. erikdarlingdata commented on Sep 26, 2026

    @erikdarlingdata
    OwnerAuthor

    Claude posting for Erik Darling

    Closing: everything on this issue has shipped, except two items that wait on data or time. Those are split into their own issues.

    Shipped

    Split out

    Not carried forward

    • The self-hosted CPU proxy: its population is 0. pg_stat_kcache is absent on 240 of 240 (server, database) pairs (batch D, D6). If that extension appears on a measured target, this becomes its own issue.
    • An Aurora peak-memory bar: none is set. The 2026-09-25 read made a bar conditional: "If a finding needs a bar, measure the distribution first" (p50 and p99 of shape peak memory against instance memory). File it if a finding needs one.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    client-siteOwned by the client-site agents (other laptop). Local sessions never pick these up.enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions