Skip to content

PostgreSQL sessions family: an operator's connection-saturation card now divides pg_stat_database.numbackends — the universal one-minute level, summed over databases at one instant, peaked over the window — by the ceiling, and names the exception-capture peak as its stated fallback; both readings ride the fact, the bars do not move (partial #3691, lane 25) - #3782

Merged
erikdarlingdata merged 2 commits into
devfrom
feat/3691-pg-saturation-numbackends
Sep 20, 2026

Conversation

@erikdarlingdata

@erikdarlingdata erikdarlingdata commented Sep 20, 2026 •

Copy link
Copy Markdown
Owner

What does this PR do?

Headline mechanism. A PostgreSQL operator's connection-saturation card now divides the right number by the ceiling. PG_CONNECTION_SATURATION's numerator moves from pg_session_states.total_sessions — the exception-capture peak — to pg_stat_database.numbackends (V133, #3716): the collector's one-minute LEVEL of backends attached to a database, summed over the databases at one collection_time, then the window's peak of those instants. Where the store does not carry the column yet — a pre-V133 store, or one mid-migration — the capture peak is used exactly as v1 did, and the fact says which (numerator_source). Both readings ride the metadata whenever both exist; Value is whichever decided; no bar moves. Lane 25 of #3691.

PARTIAL for #3691 — do not close. Completes the consumer half of this checklist line, quoted verbatim: numbackends on pg_database_stats — "(#3716, V133 — stored; the saturation numerator switch is a consumer lane once the column has data)". The column has ~20 h of fleet rows as of 2026-09-20 13:00Z.

The lie it stops telling

v1 had one numerator and it was an exception table's. pg_session_states stores a capture only when some session tripped a rule, so its peak total_sessions is a peak over the minutes that had something to report: biased HIGH on a server that captures often, blind on one that never does — the fleet's captures per server ran 59 – 2,100 over 7 days, wildly uneven — and it counts PostgreSQL's own database-less background processes, which hold no max_connections slot. Design §3.6 wanted numbackends for this from the start; the rung landed yesterday. numbackends is universal (every server, every minute), a level (never differenced, stats_reset-agnostic), and attached-to-a-database backends only.

What changes, and from which table

Darling/PerformanceMonitor.Darling.Analysis/PgTargetFactCollector.Sessions.cs — one new read, PgTargetNumbackendsPeakSql over pg_database_stats (a collector table already in the FROM/JOIN whitelist), three CTEs: instants = SUM(numbackends) … GROUP BY collection_time; shape = how many instants the window holds and how many carried the column (COUNT(*) FILTER (WHERE numbackends IS NOT NULL) — coverage in INSTANTS, not rows); peak = the largest sampled instant. LEFT JOIN … ON true so the shape row survives a window with no sampled instant. No LAG, no stats_reset — a level is never differenced. It runs on the same connection, under the same deadline and degrade, after the ceiling lookup, so a window this family would not grade costs no third query (pinned by code order).

The numerator decision is a pure public function, ChooseSaturationNumerator(databaseStatsSamples, numbackendsSamples) → (UseNumbackends, Partial): no sampled instant → capture peak, not partial (a pre-V133 store — nothing the column could have covered); coverage ≥ NumbackendsCoverageFloor (½) → the level; SOME sampled but under half → capture peak with numbackends_partial = 1 (a store mid-migration, or a collector restarted onto the rung inside the window — a peak over the sampled half of a window is the capture-peak lie in a new coat).

Metadata, stamped on every saturation fact: numerator_source (1 = numbackends, 0 = capture peak), numbackends_partial (0/1), numbackends_samples, database_stats_samples; and whenever at least one instant carried the level — INCLUDING the partial fallback — peak_numbackends and numbackends_peak_age_s. peak_total_sessions stays under its v1 name whichever numerator decided: #3713's compare banding reads it, and the state breakdown (active / idle in transaction / other) is the capture's and only the capture's. peak_age_s stays the capture peak's age; the level's instant has its own key.

Two instruments, two denominators, stated. The idle-in-transaction SHARE (peak_idle_in_transaction_share — lane 3's amplifier gate and the graph edge's) keeps the capture's own total_sessions as its denominator. numbackends has no state; a share of it would be a count from one instrument over a count from another. The collector doc, the scorer doc and the advice all say so.

PerformanceMonitor.Analysis/PgTargetScorer.Sessions.cs — constants and prose only. The eight metadata-key constants, NumbackendsCoverageFloor = 0.5 (definitional — the RedactedShareMajority shape: a column arrives with a rung and from then on every row has it, so a real store reads ~0 or ~1 and there is nothing to calibrate), and the doc updates below. ScoreSessionsFact is untouched: it grades saturation_ratio between the same bands from either numerator.

PerformanceMonitor.Analysis/PgTargetAdvice.Sessions.cs — the numerator names its instrument. When numbackends decided: "At the window's peak minute, 1.5 h before the window's end, 93 client backends were connected against 97 usable connections — pg_stat_database.numbackends summed over every database at that one instant, against max_connections 100 minus superuser_reserved_connections 3 — that is 93 / (100 − 3) = 96%.", headline "Connections peaked at 96% of the usable ceiling (93 client backends of 97) — …"; the capture is introduced as what it is — "State breakdown at the peak session capture (40 sessions, 2 h before the window's end): 15 active, 5 idle in transaction, 20 other …" — with one sentence saying the numerator is the level, not that capture, and why; the coverage sentence names how many of the window's samples carried it; the capture-only caveat ("reads a few points high") is NOT claimed of the level; the PARKED sentence and the lever sentence say "of the peak capture". When the capture peak decided: lane 3's block byte for byte plus ONE sentence — "The numerator is the peak of the session captures (exception-driven; numbackends not yet collected on this store)." — or, mid-migration, "… pg_stat_database.numbackends is present on only 101 of the window's 241 pg_database_stats samples — a store mid-migration — so the level is not used until at least half of them carry it; over the samples that do, it peaked at 93." The static block names both instruments and the rule between them and still claims no figure.

The lineage shape of every bar

Bar Value Lineage
ConnectionSaturationWarning / Critical 0.8 / 0.9 unchanged; measured (2026-09-19, §A3: per-capture sessions/ceiling per-server p99 median 0.021, fleet max 0.103 over 7 d × 50 Aurora PostgreSQL clusters). The measured note gains its second sentence from the coordinator's C1 read (production store, 13:24Z 2026-09-20, ~20 h of V133 rows): SUM(numbackends)/ceiling per-server p50 median 0.8 %, p99 median 1.8 %, fleet max 10.0 % (one cluster); the capture peak over the SAME window p99 median 2.0 %, fleet max 10.0 % — the two numerators agree within ~10 % where both exist, from 1,056 numbackends samples per server against a median 24 captures. The bars 0.8 / 0.9 stay engine-defined and deep in the measured empty interval; threshold_lineage stays 1. Today's 24 h coverage is 86 % on every server (the pre-V133 hours), so the ½ floor passes now and reads 100 % from tomorrow.
NumbackendsCoverageFloor 0.5 definitional (a majority), not a bar the fact is graded on; not stamped — the coverage figures ride the fact instead
IdleInTransactionShareBar 0.25 unchanged, measured — and still read over the CAPTURE's total_sessions, on purpose
everything else in the file — untouched

The C1 read landed after open (second commit): the scorer's ConnectionSaturationWarning block, the class summary and the advice's trailing clause carry the 2026-09-20 numbers; the fallback arm's text is unchanged (byte-identity pin still holds).

Co-fire gating

Unchanged. The three saturation amplifiers (parked share, CPU fired, offered-vs-delivered) read the same keys they did; the parked-share arm's gate is the capture's share. The graph edges (PgTargetRelationshipGraph.Saturation.cs) are untouched. A saturation fact at the same ratio scores and amplifies the same whichever numerator produced the ratio (pinned).

What it does NOT do

  • Does not speak when the window has no session capture. A quiet OLTP pool that never trips a capture rule still gets no saturation fact, even though numbackends could now give it a numerator — the state breakdown, the redaction verdict and the newest-capture figure all come from the captures, and emitting without them changes what the card says. That is the honest NEXT step (a level-only card shape), stated in the collector doc, not this one.
  • Does not build a ratio under a redacted majority. PG_MONITORING_PERMISSIONS still replaces the ratio when the login could not see state, although pg_stat_database is unprivileged and numbackends is readable there. Same reason: the family's other claims need the captures. Reported below as the coordinator's call.
  • Does not move a bar, add a key to PgTargetFactKeys, touch the graph, the scorer's grading, FactScorer, FactAdvice, any root partial, PgSessionStatesCollector, PgDatabaseStatsCollector, or a rung (StorageVersion at V135 on dev). No production read.
  • A SQL Server target's pass is byte-identical by construction (the read lives inside CollectSessionFactsAsync, behind the pg_sessions source).

Two claims from the rung's doc, checked on a live engine

Measured on the throwaway PG18 store (SELECT datname, numbackends FROM pg_stat_database; pg_stat_activity grouped by backend_type):

  1. The shared-relations row's numbackends is 0, not the NULL pg_database_stats stores numbackends and pg_wait_sampling stores sampled_ms (Darling V133), so a PostgreSQL operator's saturation fraction has a numerator and the sampler arm's wait rate has its duty-cycle denominator (partial #3691) #3716's doc expected ("NULL where the engine reports none (the shared-relations row)"). pg_stat_get_db_numbackends(0) reports 0 on PG18. Harmless to this read — 0 sums as nothing, and NULL would too — and the collector's own version audit is right that the column exists; only the row-contract sentence is off. The collector doc here says what was measured; the live fixture plants the shared row as 0. Reported for the coordinator.
  2. "Client-only" is approximately right, not literally. darling read 2 backends with ONE client connected: the TimescaleDB per-database scheduler worker counts. numbackends counts backends ATTACHED TO A DATABASE — client backends plus autovacuum / parallel / logical-replication / extension workers while inside one — and none of the database-less processes (checkpointer, walwriter, bgwriter, launchers, physical WAL senders, I/O workers) the capture's total_sessions counted. Workers hold no max_connections slot either, so the ratio can still read a hair high during a vacuum or a parallel plan — the safe direction, and far less of it than before. The collector doc and the advice state it that way rather than as a backend_type filter the view does not offer.

Test plan

Darling/Darling.Tests/PgTargetSessionsTests.cs (+4 unit facts / theories, 1 existing pin re-counted 2 → 3 commands and renamed, the existing e2e extended with a fourth server):

  • ChooseSaturationNumerator_… (6 rows): (0,0) and (241,0) → capture, not partial; (241,241) → level; (240,120) → level (half IS the floor); (240,119) and (241,101) → capture + partial; the floor is the scorer's one constant.
  • TheLevelStamps_PutBothReadingsOnTheFact_AndValueFollowsWhicheverDecided_WhileTheBarsDoNotMove: all sampled → Value 93/97, peak_numbackends 93 AND peak_total_sessions 40, idle share 5/40 (the capture's), ScoreBase 1.0 / lineage 1 / severity 1.0 with no arm matched through the real ScoreAll; partial → 40/97, source 0, partial 1, the unused 93 still shown, scores 0; none → 90/97, no level keys; the six key strings by name.
  • ComposeSessions_NamesTheLevelAsTheNumerator_…: the headline and the first sentence by exact string; the capture introduced at its own instant and count with the unchanged split; "The numerator above is the level, not that capture"; the coverage sentence; the pinned measured clause AND its second sentence; the capture-only caveats absent; over the parked bar the share and lever say "peak capture".
  • ComposeSessions_StatesTheCapturePeakAsTheNumeratorWithItsReason_…_AndAPreV133FactIsByteIdentical: lane 3's sentences byte for byte plus the one fallback sentence; Assert.Equal(preLane, block) — a fact with NO level keys composes the same block as one stamped "capture peak, nothing sampled"; the mid-migration sentence by exact string; the static block's two instruments.
  • TheSessionReads_AreThreeQueriesOverCollectorTables_… (was TwoQueries): the new const in AllSql; FROM pg_database_stats only; SUM(numbackends) BEFORE GROUP BY collection_time and exactly one SUM(; the peak ORDER BY numbackends DESC, collection_time DESC; no LAG( / stats_reset; the IS NOT NULL coverage and peak filters; LEFT JOIN peak AS p ON true; the level read AFTER the ceiling lookup in code order; ChooseSaturationNumerator(level.…) and the three key constants named in code; ["peak_total_sessions"] = peakTotal retained; 3 commands / 3 deadlines.

Gated e2e (the existing APlantedSeries… test, fourth server …-level): 25 h of span plus every minute of the 4 h window as the V133 collector writes it — appdb 20 + reports 10 + the shared row's 0 = 30, and ONE instant 90 min before the end at 60 + 33 = 93 — beside 48 captures peaking at 40 (15 / 5 / 20). The collector alone: Value 93/97, source 1, partial 0, peak_numbackends 93 at 5,400 s, 241 / 241 instants, peak_total_sessions 40 at 7,200 s, the split, idle share 5/40; scored 1.0 / 1.0 / lineage 1. Through the REAL analyze_server anchored at the window's end: the card at 1.0 with the exact headline, the first sentence, the capture-breakdown sentence, the coverage sentence, no fallback sentence; get_analysis_facts shows source 1, 93 and 40 side by side. Then the same store mid-migration (UPDATE … SET numbackends = NULL on every instant older than the last 100 min → 101 of 241): 40/97, source 0, partial 1, peak_numbackends still 93, the mid-migration sentence. Then NULL everywhere: 40/97, source 0, no level key, the not-collected sentence — v1's reading. The busy server (the shared planter's NULL column) additionally asserts source 0 / 0 of 241 / no level key, and lane 18's 31-day e2e (NULL column) passes unchanged — the byte-identity claim, executed.

Executed here, not only compiled: a net10.0 harness (AssemblyName=Darling.Tests, 20 PgTarget census / hygiene classes incl. PgTargetFactCollectorTests, PgTargetSharedSwitchRoutingTests, PgTargetThresholdLineageTests, PgTargetMeasuredLineageTests, StoreSqlClockDisciplineTests, DocCommentHygieneTests, RepoFileAdoptionTests, FleetIdentifierScrubTests, FactCollectorCommandTimeoutTests, FactSourceRegistryTests, LiveCleanupConversionRatchetTests, LivePostgresCollectionHygieneTests, PgTargetBetweenWaves*, PgTargetWaitTests, PgTargetNamedHopsTests, CollectionCaveatsTests) under <worktree>/obj/mactest/ against a throwaway timescale/timescaledb:2.28.1-pg18 on Docker context desktop-linux (role and database darling, migrated by the fixture to V135), torn down after: 283 / 283 green with DARLING_TEST_PG set — every gated e2e in the set ran (183k pg_database_stats inserts on the store), including this lane's and lanes 3 / 14 / 18's. First execution of every other census is CI's Windows job.

Builds (-c Release -p:EnableWindowsTargeting=true): PerformanceMonitor.Analysis, PerformanceMonitor.Darling.Analysis, PerformanceMonitor.Darling.Service, deprecated/Dashboard.Tests — 0 warnings; Darling.Tests — one pre-existing xUnit2013 (ConfigChangeTraceAnchorTests.cs:94); Lite.Tests — the two pre-existing (LogTailOverlapThresholdPinTests.cs:95 xUnit2000, DefaultTraceEventsCollectorDefinitionTests.cs:211 xUnit2013). None in a file this PR touches.

Which component(s) does this affect?

  • Lite
  • Darling
  • Lite Tests
  • Darling Tests
  • SQL collection scripts
  • Documentation
  • Full Dashboard (deprecated)
  • CLI Installer (deprecated)

How was this tested?

Above. Throwaway container only; no production store or fleet server was read.

Checklist

  • I have read the contributing guide
  • My code builds with zero warnings (dotnet build -c Debug)
  • I have tested my changes against at least one SQL Server version (PostgreSQL-target surface; the SQL Server pass is byte-identical by construction — the read sits inside the pg_sessions family)
  • I have not introduced any hardcoded credentials or server names

Partial for #3691 (lane 25). CHANGELOG deliberately untouched — the coordinator splices this wave's entries. Not armed (review-dark window).

…ceiling — the universal one-minute level, summed over databases at one instant, peaked over the window — and names the exception-capture peak as its stated fallback; both readings ride the fact, the bars do not move (partial #3691, lane 25)
…ends/ceiling read on 2026-09-20 (p99 median 1.8 %, fleet max 10.0 % over 20 h × 50 clusters) agrees with the capture peak within 10 %; the bars stand (partial #3691, lane 25)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant