PG_BLOAT_TREND: a PostgreSQL operator learns which table is actually accumulating space over fourteen days — growth of the estimate behind a size floor, never the spot percentage that reads 86% on a 2 MB table (partial #3691)

[erikdarlingdata]

What does this PR do?

Headline mechanism. The pg_bloat family of the PostgreSQL-target analysis engine (#3691 lane 13, design §3.4): PG_BLOAT_TREND over the hourly pg_table_bloat_stats samples and PG_INDEX_BLOAT_TREND over the daily pg_index_bloat samples. Both grade fourteen-day GROWTH of the bytes estimate — latest usable sample minus earliest usable sample inside a lookback that ends at the analysis window — behind a 64 MiB size floor, and neither ever grades a spot percentage. A PostgreSQL operator now learns which table (and which index) is actually accumulating space over the last two weeks, by how many bytes and what fraction of where it started, over how many samples, whether autovacuum's backlog on the same table is the cause — and, in so many words, that the 86 % a screen shows for a 2 MB table is not a reason to do anything. PARTIAL for #3691 — do not close. Checklist line completed: PG_BLOAT_TREND.

Why a trend and never a spot value — the measured reason

The fleet calibration pass of 2026-09-19 (14 days × 50 Aurora PostgreSQL clusters of the dogfood fleet) read bloat_pct_estimate: per-server median 12.9 %, but fleet p75 71 %, p90 86 %, max 99.8 % — driven by small heaps (median heap 75 MB) where the column-width arithmetic is noise against a few thousand pages. Index est_bloat_pct read p50 46 %, p99 88 % with a median index size of 16 kB, most indexes skipped. Fourteen-day growth per table: p50 zero bytes on most servers; p99-table median 5.6 MB / 6.3 %; fleet p90 874 MB / 120 %; max 6.8 GB / 523 %. A percentage graded without a size floor would page on most of the fleet; a spot percentage graded at all would turn PgMigrations.V85's recorded trap (88.59 % reported against a true 0.50 % when the monitoring role could not see pg_stats) into a finding. The design (§3.4) predicted exactly this; the measurement confirmed it; the family is built on it.

What each fact concludes, and from which table

PG_BLOAT_TREND — pg_table_bloat_stats, hourly. Usable rows are estimate_unavailable = false (a NULL flag reads as unavailable, never available). For every table with a usable estimate in the lookback, latest beside earliest; only tables whose latest heap_bytes ≥ 64 MiB are considered; growth = latest − earliest bloat_bytes_estimate. One fact per server, ObjectName = the worst table, Value = its growth bytes, the top three by name in metadata (growth_bytes_<schema.table>, growth_pct_<…>, dead_tuples_<…> — name in the key, the wait profile's contrib_ shape, because metadata is doubles). Population counts ride on the fact (tables seen / estimable / over the floor / over the line) so get_analysis_facts shows why nothing graded when nothing did.

PG_INDEX_BLOAT_TREND — pg_index_bloat, daily. Usable rows are skipped_reason IS NULL (the collector writes the estimate NULL under exactly that condition — V114); growth is on est_reclaimable_bytes (the figure the reads rank on, #2561 — bytes, never a percentage); floor on latest index_bytes ≥ 64 MiB; an index needs three samples in the lookback before it is ranked (the daily cadence gives at most fourteen). The fact carries skipped_share and the count of DISTINCT skip reasons (counts, never the prose — get_pg_index_bloat shows them) and whether pgstattuple was seen installed. ObjectName is schema.table.index so the graph can recover the parent.

The withheld shapes are facts too, at base 0, each carrying estimate_available = 0 and an unavailable_reason code: every row estimate_unavailable (the #2542 grant); every index skipped, split on whether pgstattuple is installed (the exact route the skip reasons point at — open or closed); estimable objects but none over the size floor; indexes over the floor but none with three samples yet. Each has its own advice card, and every one of them says the same sentence: a withheld estimate never reads as zero bloat. No rows in the lookback at all → no fact, never a zero.

Lookback, not window. The pass window is hours; bloat moves in days. The reads run from TimeRangeEnd − 14 days to TimeRangeEnd — a longer span than the pass, the way the baseline provider reads 30 days behind its window — and the fact stamps lookback_days, its own samples and span_hours, and samples_in_lookback (distinct collection times). It never borrows the one-minute coverage fraction and divides by nothing in ObservedDurationMs: growth is a LEVEL differenced across the object's own samples.

No anomaly detector, stated in the collector's doc: the trend is computed in-window against the object's own earlier value, and an hourly/daily series would put two samples a fortnight into an hour-of-week bucket.

The lineage shape of every bar

Bar	Value	Lineage
Growth line, bytes arm (WARNING at 0.5)	256 MiB	measured — fleet p90 of worst-table 14-day growth over 14 days × 50 Aurora PostgreSQL clusters of the dogfood fleet, 2026-09-19
Growth line, fraction arm	25 % of the earlier estimate	measured — same read; well under the p90 (120 %) so the bytes arm discriminates. An earlier estimate of zero passes this arm trivially and the fact says the percentage is not computable
CRITICAL (1.0)	1 GiB	measured (empty interval) — fleet max 6.8 GB on one table, nothing else approached a gigabyte
Size floor (heap / index)	64 MiB	measured — median heap 75 MB, median estimable index 16 kB; the small objects are exactly where the 71–99.8 % spot figures live
Lookback	14 days	measured — the calibration pass's own window, so the line and the span agree
Index sample minimum	3	unmeasured — chosen, not measured; calibrate against pg_index_bloat before the next release
Amplifier boosts (0.3 / 0.2 / 0.3)		unmeasured — chosen, not measured; calibrate against the fleet's co-fire rates

Growth of a bytes estimate is engine-neutral by construction, so the Aurora population stands in for stock PostgreSQL — stated in the scorer's doc, not assumed. A fact graded through the measured bars stamps threshold_lineage = 1; an index fact held back by the chosen sample minimum stamps threshold_lineage = 0 — the flag names whichever bar decided. The scorer re-applies the floor and the line to the fact's metadata (a hand-built fact cannot bypass the collector's bind), and the collector binds the SAME constants ($4–$8) into the SQL so the count of tables over the line and the grade cannot drift. PgTargetThresholdLineageTests passes over the new partial.

The co-fire gating

• PG_BLOAT_TREND ↔ PG_AUTOVACUUM_BACKLOG, same table. Bloat is the DAMAGE whose cause is §3.1's backlog. The predicate intersects the backlog fact's one table (ObjectName) with the set the bloat fact names — PgTargetScorer.BloatAndBacklogShareATable is the single definition the edge (both directions) AND the 0.3 amplifier use, so they cannot disagree. When the backlog fact carries no name the intersection cannot be tested and it falls back to co-presence; the description says so. Both directions, for the vacuum mesh's reason: which of the two leads is not fixed (a 900 MB growth outranks a table just past its line; a 40× backlog outranks a growth just over 256 MiB), and one direction would leave the other fact rooting a second story about the same table. The backlog → trend edge is declared in the bloat partial (it exists only when the trend does); the vacuum mesh's pins are untouched.
• PG_INDEX_BLOAT_TREND → PG_BLOAT_TREND, same parent table (the index name's schema.table prefix against the table trend's named set), plus the matching 0.3 amplifier.
• PG_XMIN_HOLD co-presence lifts the table trend 0.2 (a held horizon makes the dead tuples unreclaimable, so the heap grows around them). No edge to the hold: it reaches bloat through the backlog, which already meshes with it.
• No config co-fire: fillfactor is per-object and no server knob makes a table bloat. Under the line, amplifiers never lift a fact (FactScorer skips amplifiers at base 0 — pinned).

Advice (OtterTune doctrine)

Value-stated from the fact — the object, its measured size, the estimate at both ends, growth in bytes and (where computable) percent, samples and span, dead/live tuples now, the next two tables by growth, the population counts, the co-fires by name. The investigation says the spot estimate "was NOT graded" and why, with the measured figures. The remediation opens with "The trend is the finding; the percentage is not — nothing here is a reason to run VACUUM (FULL) on a Tuesday" and orders the levers with a counter-objective each: the vacuum side first (the same-table backlog card, or the table's own autovacuum_vacuum_scale_factor reloption read from get_pg_autovacuum_health — more vacuum I/O), pg_repack second (online; an extension plus a full copy's worth of I/O and disk), VACUUM (FULL) last (ACCESS EXCLUSIVE lock for the rewrite, the table's size again in free space). The index card points at get_pg_index_usage FIRST (an unused index's bloat is a reason to drop it, not rebuild it), then the table's card if it co-fired, then names the concurrent rebuild with its cost (a second copy on disk, the build I/O, a transaction that holds xmin). D8: no CREATE INDEX, no REINDEX statement against any object — the lever is named, never emitted; RemediationTsql is null. next_tools were already wired by plumbing (get_pg_table_bloat / get_pg_autovacuum_health; get_pg_index_bloat / get_pg_index_usage) — the brief's get_pg_autovacuum_stats does not exist; get_pg_autovacuum_health is the read.

What it does NOT do

No anomaly detector, no baseline, no AnomalyThresholds constant, no rung (StorageVersion stays V132), no new key or source (both keys and pg_bloat came with #3715), no shared-file edit (FactScorer, FactAdvice, InferenceEngine, RelationshipGraph, the root partials, PgTargetFactKeys, PgTargetToolRecommendations untouched). No per-table fillfactor reasoning. No Aurora suppression (bloat is engine-neutral; nothing to suppress). No view of TOAST or of index bloat via pgstatindex measurements (the store's exact-measurement columns stay unread — the family reads the estimate route only, and says when the exact route is open). A SQL Server target's pass is byte-identical: every new arm is behind PgTargetSources.BloatSource / the two PG_*BLOAT_TREND keys, which a SQL Server fact cannot carry; the routing pins pass unchanged.

Boundary

Declared files: the four *.Bloat.cs stubs and the two new test files. One census pin moved by design: PgTargetSharedSwitchRoutingTests.FactAdvice_DelegatesEveryPgKey… — the lane-13 line goes Null → NotNull (the plumbing PR's own instruction: "each lane moves its own line"), plus the IndexBloatTrend sibling line.

Test plan

New: PgTargetBloatTests (35 tests) — the constants by value; the table base across both arms of the line, the earlier-zero case, the ramp, the cap, the heap floor and a shrink; the withheld shapes score 0 with their reason and no lineage flag; the index sample minimum with threshold_lineage = 0; the 64 MiB index floor; BloatNamedObjects / parent-table parsing / the same-table predicate including the nameless fallback; the four amplifier shapes (same table, different table, hold, under-the-line never lifted); the three story shapes (same table one story either leading; different tables two stories; index → table and index alone); the static blocks exist and equal FactAdvice.GetForFactKey; the composed table card's headline VERBATIM, its value sentences, lever ORDER and counter-objectives, the different-table backlog sentence; the earlier-zero and under-line headlines; all six withheld cards by headline and remediation fragment; the index card verbatim; the two SQL consts by text (unavailable filter, floors, the bound line, no ranking on percentage, the population LEFT JOIN). New: PgTargetBloatLiveTests (DARLING_TEST_PG, [Collection("live-postgres")], LiveStoreCleanup) — the brief's exit criterion: 14 d hourly for one 1 GiB table growing 100 MB → 500 MB with a pre-lookback zero row and an estimate_unavailable 5 GB garbage row planted as traps, twenty 2 MB tables at 90 % "bloat" every six hours; ten daily index samples 50 MB → 950 MB beside five 16 kB skipped indexes with two reasons; a same-table backlog. Asserts exactly one PG_BLOAT_TREND naming public.big with earlier = 100 MB (both traps excluded), 336 samples, population 21 / 21 / 1 / 1, no small table named; one PG_INDEX_BLOAT_TREND with skipped share 5/6 and 2 reasons; bases 0.59375 / 0.919 with threshold_lineage = 1; and through the REAL analyze_server, ONE story PG_INDEX_BLOAT_TREND → PG_BLOAT_TREND → PG_AUTOVACUUM_BACKLOG, its headline verbatim, PostgreSQL next_tools for every hop, get_analysis_facts showing both under pg_bloat.

EXECUTED here through a net10.0 harness (AssemblyName=Darling.Tests, the 35-file PgTarget census set plus both new files and the vacuum live class) against a throwaway timescale/timescaledb:2.28.1-pg18 on Docker context desktop-linux, migrated to V132 by the fixture, torn down after: 459 / 460 green with all 8 gated e2es running — mine, the plumbing exit (analyze_server → empty on a target with no bloat rows: the family emits nothing without data), lane 4's vacuum mesh (still one story — no bloat fact there), lane 9's anomalies. The one red is the harness csproj tripping DocCommentHygieneTests.TheDerivedProjectListCoversEveryProjectInTheTree (the artifact #3665 / #3715 recorded; the harness is deleted). Censuses green in that run: PgTargetThresholdLineageTests, PgTargetFactCollectorTests (AllSql closure, dialect, FROM/JOIN whitelist, the lane-13 marker), PgTargetSharedSwitchRoutingTests, PgTargetMcpSurfaceTests, PgTargetKnobsTests' D8 sweep, StoreSqlClockDisciplineTests, DocCommentHygiene over the new files, LiveCleanupConversionRatchetTests, LivePostgresCollectionHygieneTests, RepoFileAdoptionTests. No file mentions an aurora_stat_* surface.

Builds, zero warnings of ours: PerformanceMonitor.Analysis, PerformanceMonitor.Darling.Service, Darling.Tests, Lite.Tests, deprecated/Dashboard.Tests (-p:EnableWindowsTargeting=true on the test projects). The one Lite.Tests warning (LogTailOverlapThresholdPinTests.cs:95 xUnit2000) is pre-existing on dev. First execution of everything outside the harness set is CI's Windows job.

Which component(s) does this affect?

• Lite
• Darling
• Lite Tests
• Darling Tests
• SQL collection scripts
• Documentation
• Full Dashboard (deprecated)
• CLI Installer (deprecated)

How was this tested?

Above. No production store or fleet server was read; the measured figures are the coordinator's calibration pass, quoted.

Checklist

• I have read the contributing guide
• My code builds with zero warnings (dotnet build -c Debug)
• I have tested my changes against at least one SQL Server version (the SQL Server pass is unreachable by every new arm; the routing pins that proved v1's byte-identity pass unchanged)
• I have not introduced any hardcoded credentials or server names

Partial for #3691 (lane 13; checklist line PG_BLOAT_TREND). CHANGELOG deliberately untouched — the coordinator splices this wave's entries.