Skip to content

PostgreSQL buffer-pressure cards carry what the cache HOLDS — the latest pg_buffer_usage capture as pool fill, the ten largest resident relations, the index/heap/TOAST split and the cold share, folded into the advice, so an operator learns whether shared_buffers is one relation or a working set (#3691) - #3795

Merged
erikdarlingdata merged 1 commit into
devfrom
feat/3691-pg-buffer-composition
Sep 20, 2026

Conversation

@erikdarlingdata

@erikdarlingdata erikdarlingdata commented Sep 20, 2026 •

Copy link
Copy Markdown
Owner

What the buffer-pressure card now says

A PostgreSQL card rooted on PG_BUFFER_CACHE_PRESSURE — the memory chain PG_BUFFER_CACHE_PRESSURE → CONFIG_PG_SHARED_BUFFERS, or the composite alone on a server whose knob is not at its default — now carries a pg_buffer_composition drill-down and a re-frozen advice block that say what the cache HOLDS beside the counters that said it is short. The composite grades hit ratio, evictions and bgwriter halts, every one a rate saying the working set does not fit; none of them says what the working set is. The latest pg_buffer_usage capture in the window does: pool fill, the ten largest resident relations with their share of the pool, dirty share and average usage count, the split between indexes, heaps and TOAST, the dirty share of the pool, and the share sitting in relations the clock sweep evicts first. Before this the card said "50 % of block requests left shared_buffers … raise it toward a quarter of RAM" and pointed at get_pg_buffer_usage; the operator had the rate and none of the composition.

Partial for #3691 — checklist line, inside the v3 line: buffer-composition drill-down (pg_buffer_usage).

The source and the shape

One read of pg_buffer_usage (V93, already a CollectorCatalog target — plumbing pinned it by name for this lane): the newest CTE picks the latest collection_time in the window and every row is from that capture, because residency is a LEVEL and the pool totals are consistent within one capture and not across two (DarlingPgBufferUsageReader's reasoning, copied). The totals — pool used/total, buffers_listed, dirty, heap/index/TOAST, cold — are aggregates over the capture in a totals CTE cross-joined onto the top-$4 rows, never a second read against a pool that has since moved. buffers_listed is at most pool_buffers_used because the collector drops relations under 8 buffers; the payload states both rather than pretending the tail is not there. Kind shares split buffers_listed by relkind — r/m heap, i index, t TOAST, everything else (including the NULL kind of a buffer from another database or a shared catalog, which the collector keeps because dropping it would understate the fill) as other — and sum to 1. The cold share is buffers in relations whose avg_usage_count ≤ 1: engine-defined (the clock sweep decrements on each pass and evicts at 0, so ≤ 1 is the next victim), and a PER-RELATION average, which the prose says.

Names are bounded in the read (LEFT(relation_name, $5) + length() for relation_name_truncated, lane 7's shape) at NAMEDATALEN − 1 = 63 — a collector-written name is never cut; the bound is the payload contract, not a privacy cut.

The pressure figures are reused, never recomputed

miss_share, hit_ratio_suppressed, evictions, buffers_alloc, observed_ms, cache_turnovers_per_hour come off AnalysisFinding.RootFactMetadata (the composite roots the memory chain) and ride under pressure so the rate and the level sit side by side. The fact's per-second quotients (evictions_per_sec, buffers_alloc_per_sec) are deliberately NOT re-published here: they are computed once in PgTargetFactCollector.Buffer.cs over observed time and are the fact's to publish (get_analysis_facts source=pg_buffer; the advice prose states them) — the counts and the observed span ride instead, and MeasurementContractCensusTests rule 6 (a per-second key's value must be a reader rate or a C# quotient) is why a copy under a per-second name is the wrong shape. CI caught the first push on exactly that, and on TsqlConventionGuardTests' member walk stopping short at an is { } v expression body — both fixed in the second commit, neither rostered. The drill-down never opens pg_database_stats, pg_io_stats or pg_write_stats — pinned over the comment-stripped code and the three SQL consts.

Nothing to show is a sentence, and WHICH sentence is read

pg_buffer_usage collects hourly and needs pg_buffercache. A window with no capture runs two more bounded reads: pg_extension_availability's latest state for pg_buffercache at or before the window's end (one row per distinct state, since the collector writes per database), folded installed/outdated-anywhere → cadence, else available → pg_buffercache_available ("one CREATE EXTENSION away"), else absent → pg_buffercache_absent ("not on this server's menu; on a managed service a parameter-group question"); and, for the cadence arm, max(collection_time) over the server so the sentence says "the last capture was N h before the window" or "this server has no capture at all". A missing availability row is reported as the cadence arm: the pass did not read a state and does not claim one. The extension read runs ONLY when there is no capture — a capture is proof the extension worked.

The advice: extended, not repeated

FactAdvice.PopulateStoryText freezes the card from facts alone at step 3.5, before drill-downs; so, lane 16's shape, the drill-down re-freezes through PgTargetAdvice.WithBufferComposition(frozen, summary) — words in PgTargetAdvice.Buffer.cs beside lane 2's buffer prose, read in the Darling partial. The investigation gains the value-stated sentence (Composition: the pool is 94 % full (15,400 of 16,384 buffers, 128 MB); \orders_idx` (index, appdb) alone holds 30.5 %; of the resident buffers, indexes hold 37.6 %, heaps 48.9 %, TOAST 7.5 % …; 10.1 % of the pool is dirty; 33.8 % of resident buffers sit in relations whose average usage count is at or below 1 — the clock sweep evicts those first. Captured at … UTC — a level, not the window's counters). The remediation, when a capture was read, gains "What the capture says about sizing": a relation holding ≥ 50 % of the pool is named before the knob (what reads it — get_pg_top_queries, the shape whose shared_blks_readdwarfs itsshared_blks_hit— and its bloat); a cold MAJORITY says a larger cache would hold pages nobody re-reads, "raising it buys little here and still pays the counter-objective named above" — lane 2's D1 counter-objective sentence is stated ONCE, pinned; a pool under 90 % full says the size is not the constraint; and on Aurora (offhit_ratio_suppressed) the composition is called the honest read where the hit ratio is stated but not graded — lane 2's wording, kept. Idempotent on the Composition: marker.next_toolsfor the pressure key gainsget_pg_top_queriesbesideget_pg_buffer_usage`.

The three prose selectors (0.5 dominant share, 0.5 cold majority, 0.9 fill) are NOT bars — nothing is graded on them, no fact carries them, they choose which sentence is said and the sentence states the number either way; their comments say "chosen, not measured" (the 2026-09-19 calibration held no pg_buffer_usage figures). No threshold, ramp or floor is added to any scorer; PgTargetScorer.Buffer.cs is untouched.

Gating

Inside the shared 0.5 gate, on the pressure key ONLY. The composite scores 0 below its concerning line and ≥ 0.5 at it (GradeArm → ApplyThresholdFormula), so "on the path" already means "fired at the incident line"; the CONFIG_PG_SHARED_BUFFERS advisory rooting alone at 0.4 on a quiet server is deliberately not enriched — the composition of a cache nothing is short of is get_pg_buffer_usage's to browse. Pinned by the root's text.

What it does NOT do

  • No new key, stub, migration, tool or scorer change; PgTargetFactCollector.Buffer.cs and PgTargetScorer.Buffer.cs untouched.
  • Does not name pg_buffercache's state when a capture exists, and does not read pg_extension_availability then.
  • Does not per-buffer the cold count — the stored figure is a per-relation average, and the payload's cold_note says so.
  • Does not schema-qualify relation names — the collector stores relname; the sentence states what is stored.
  • A SQL Server target's pass is byte-identical: the dispatch is inside the PostgreSQL-only PgTargetDrillDownCollector, the composer is called only from there, and the routing pins re-prove it.

Boundary extensions

  • Darling/PerformanceMonitor.Darling.Analysis/PgTargetDrillDownCollector.cs (+8, granted): the dispatch branch and the partial declaration.
  • Darling/PerformanceMonitor.Darling.Service/Mcp/PgTargetToolRecommendations.cs (+2): the brief names next_tools: get_pg_buffer_usage, get_pg_top_queries; the first was there, the second was not. PgTargetMcpSurfaceTests (registered tool, get_pg_ prefix, dedup) holds.
  • Darling/Darling.Tests/PgTargetKnobsTests.cs (one word): PlantConfigSnapshotAsync private → internal so the e2e plants lane 2's defaults snapshot instead of copying it.

Tests

New Darling/Darling.Tests/PgTargetBufferDrillDownTests.cs (11 tests, 10 pure + 1 gated live):

  • The three SQL consts by their own text (latest-capture join, CROSS JOIN totals, the kind and cold FILTERs, LIMIT $4, LEFT/length pair, the extension filter, <= $2), clock discipline and dialect on all three, every FROM/JOIN a collector table or CTE; the caps.
  • Source pins: dispatch on the pressure key and NOT the knob; three CommandTimeout = AnalysisCommandTimeoutSeconds; every read cancellable; RootFactMetadata + the five figure names; no counter table in code or SQL; one section; re-freeze through the composer; no DeSkew; block size by name.
  • KindLabel partition = the SQL's; ClassifyExtensionStates precedence; Share null over zero; IsTruncated only on a real cut; the fixture's shares sum to 1 with every value as arithmetic.
  • The sentence: the captured arm as ONE pinned string (the same string the e2e expects from the real pass), unnameable top relation (other database / shared catalog), ellipsis on a cut name, dropped clauses; the three empty arms word-for-word, stating no pool number, no CREATE INDEX.
  • The fold: headline kept, investigation ends with the sentence, sizing paragraph's every arm (dominant / cold majority / room / Aurora) and their absence, D1 counter-objective once, idempotent, empty arms keep the base remediation, round-trips through SerializeForStoryText/TryReadStoryText, no SQL Server nouns, no DDL, no effective_cache_size, no posture knob.
  • Live (DARLING_TEST_PG, [Collection("live-postgres")], LiveStoreCleanup.RunAsync): 3,000 hits + 3,000 reads a minute over one observed hour (miss 0.5 at 100 blocks/s → 1.0), lane 2's defaults snapshot, a 20-relation capture 30 min before the window's end and an OLDER capture 3 h earlier that must not be read ⇒ the REAL analyze_server (anchored) story PG_BUFFER_CACHE_PRESSURE → CONFIG_PG_SHARED_BUFFERS carries captured, pool_fill 0.9399, ten top_relations in order with the other-database row unnamed at rank 5, kind shares summing to 1, cold_share 0.3383, pressure.miss_share 0.5 with the counts and observed span, the pinned sentence once and equal to note, the sizing paragraph, both next_tools; a second server with the same shortage, no capture and pg_buffercache available ⇒ pg_buffercache_available, null captured_at/pool_fill, empty top_relations, the extension sentence in the frozen investigation, no sizing paragraph.

Executed here before CI: all 10 pure pins green on first run; the census set (PgTargetFactCollectorTests, PgTargetSharedSwitchRoutingTests, PgTargetMcpSurfaceTests, PgTargetThresholdLineageTests, PgTargetMeasuredLineageTests, PgTargetV3PlumbingTests, StoreSqlClockDisciplineTests, DocCommentHygieneTests, RepoFileAdoptionTests, LivePostgresCollectionHygieneTests, LiveCleanupConversionRatchetTests, FleetIdentifierScrubTests, lane 2's PgTargetKnobsTests, lane 16's PgTargetDeadlockDrillDownTests) 221/221 through a net10.0 harness inside the worktree, then 221/221 with 0 skipped against a throwaway timescale/timescaledb:2.28.1-pg18 on desktop-linux (V135) — the new e2e AND lane 2's and lane 16's e2es green with the root dispatch changed; harness and container deleted. CI's Windows job is the first execution of everything else.

Which component(s) does this affect?

  • Lite
  • Darling
  • Lite Tests
  • Darling Tests
  • SQL collection scripts
  • Documentation
  • Full Dashboard (deprecated)
  • CLI Installer (deprecated)

How was this tested?

Builds: PerformanceMonitor.Analysis, PerformanceMonitor.Darling.Analysis, PerformanceMonitor.Darling.Service, Darling.Tests, Lite.Tests, deprecated/Dashboard.Tests — 0 warnings, 0 errors (test projects with -p:EnableWindowsTargeting=true). Pins and the live e2e executed locally as above; PostgreSQL 18 store. No production store or fleet server was read.

Checklist

  • I have read the contributing guide
  • My code builds with zero warnings (dotnet build -c Debug)
  • I have tested my changes against at least one SQL Server version (PostgreSQL-target change; a SQL Server pass is untouched by construction — the routing pins hold)
  • I have not introduced any hardcoded credentials or server names

Partial for #3691 (v3, lane 29). CHANGELOG deliberately untouched — the coordinator splices this wave's entries.

…st pg_buffer_usage capture as pool fill, the ten largest resident relations, kind and cold shares, folded into the advice (partial #3691, lane 29)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant