The temp-spill family of a PostgreSQL analysis pass: spilled bytes per second of observed time from the reset-aware pg_database_stats difference, work_mem graded only on that evidence and lifted to the incident line only when the spill fires, the offending statements in the drill-down; the same read emits PG_TPS with a half-window trend and the counter-derived PG_DEADLOCK_RATE on the alert band's measured tiers (#3542 lane 6)

[erikdarlingdata]

What does this PR do?

Lane 6 of #3542 (v1 step 6, design §3.9) — PARTIAL, do not close. A PostgreSQL operator now learns, from one analyze_server call, that sorts and hashes are spilling to temp files, how fast, in which database, against what work_mem, and which statements are doing it — and work_mem advice appears only when that evidence exists.

The family, and where each number comes from

PG_TEMP_SPILL (pg_temp) — spilled bytes per second of OBSERVED time (context.ObservedDurationMs, never the nominal window), from pg_database_stats.temp_bytes / temp_files differenced per database_name series with the reset-aware shape taken verbatim from DarlingPgDatabaseReader.PgDatabaseSql — ROW_NUMBER() OVER series > 1 AND stats_reset IS DISTINCT FROM LAG(stats_reset), GREATEST(raw, 0) — so a pg_stat_reset() mid-window is clamped AND reported (stats_reset_count, counter_rewind_count ride the fact). That comment records the NULL→timestamp first-reset trap; this is the fifth site of the shape and re-derives nothing. The fact names the database that spilled the most with its share, and carries work_mem_bytes / max_connections read off the config facts emitted a moment earlier so the advice can do arithmetic with the host's numbers.

CONFIG_PG_WORK_MEM — evidence-gated (D5), made structural. It is not in ConfigAdvisoryRoots (verified; plumbing left it out). Its base is 0 unless the Database read stamped temp_spill_bytes_per_sec onto it at or above the spill floor, and then it is ConfigAdvisoryBase (0.4), lifted to exactly the 0.5 incident line by the spill co-fire (KnobCoFireBoost, 0.4 × 1.25 — the shape shared_buffers and max_wal_size take). No spill fact → 0. Spill below the floor → 0. The VALUE is never graded: there is no right work_mem without the workload.

PG_TPS (pg_database) — context, base 0: xact_commit + xact_rollback per observed second, rollback share stamped, and per the coordinator's addition tps_trend (second-half rate minus first-half rate, each over its own observed time apportioned by the intervals that fell in it; stamped only when both halves were observed) plus tps_first_half / tps_second_half, for lane 3's parked offered-vs-delivered co-fire.

PG_DEADLOCK_RATE (pg_database) — the COUNTER's deadlocks per observed hour, never a pg_deadlocks row count (the rate SQL is pinned DoesNotContain("pg_deadlocks")). The exemplar count from the log capture is a separate read behind its own degrade and rides as exemplar_count beside counter_count, so the advice can say "the engine counted 99; 3 were captured from the log" and why the two differ (design §6 C). 25/hr grades Critical.

PG_HIT_RATIO is declared and deliberately NOT emitted — the hit ratio is an arm of lane 2's PG_BUFFER_CACHE_PRESSURE; a second fact over the same counters would double-count (D2). The Database.cs doc says so.

Lineage of every bar (#3538 A5/A7)

• TempSpillConcerningBytesPerSec = 1 MiB/s, TempSpillCriticalBytesPerSec = 10 MiB/s — unmeasured, chosen, not measured; calibrate against the per-server temp_bytes/s distribution over pg_database_stats before the next release; the fact carries threshold_lineage = 0. Below the floor the fact is 0, not the formula's fraction, so a trickle never arms the knob. The baseline-relative z the design names is lane 9's; this is the v1 spot fact and says so.
• DeadlockWarnPerHour = 5, DeadlockCriticalPerHour = 20 — measured: ServerHealthThresholds.DeadlockWarnPerHourDefault / DeadlockCriticalPerHourDefault (A single resolved deadlock bands a server Critical: DeadlockSeverity is count > 0 with no magnitude tier, so Critical stops discriminating on any OLTP fleet #3368, 14 days × 43 servers, 14,448 server-hours), the same tiers the fleet card already bands a PostgreSQL server's FleetPgDeadlockSql difference on. Repeated by value because PerformanceMonitor.Analysis references nothing (the PostgresOutagePredictorThresholds argument); PgTargetTempTests pins the pairs equal, the FactScorerTests idiom.
• WorkMemBootValMb = 4 — engine-defined (the compiled boot_val), used only for "the shipped default" in prose.
• Boosts (TempCauseBoost 0.3, KnobCoFireBoost by reference) — unmeasured, marked.
• Every rate divides by observed time; no absolute-ms gate anywhere.

Why the spill is also amplified by the knob having co-fired

Found by executing the pins: a spill exactly at the floor is 0.5 and the co-fired knob is exactly 0.5; the inference engine orders entry points by severity and breaks ties by emission order, and the config family is emitted first — so the knob rooted, consumed itself, and the spill rooted a second card. With the knob co-fire boosting the spill (≥ 0.65 whenever the knob fired) the spill always leads, the walk is PG_TEMP_SPILL → CONFIG_PG_WORK_MEM, and the knob is the leaf D5 says it is. A temp-writing PG_BAD_ACTOR_* of the window is the second corroboration.

The one edit outside this lane's declared partials, and why

PgTargetScorer.Config.cs gains one case PgTargetFactKeys.ConfigWorkMem: return ScoreConfigWorkMem(fact); arm (body in PgTargetScorer.Temp.cs) and its doc sentence updated. It is required, not convenient: FactScorer.ScoreAll never amplifies a base of 0, and CONFIG_PG_WORK_MEM's source is pg_config, so its base can only come from ScoreConfigFact — lane 2's own doc comment there said work_mem is "lane 6's arm … falls to the default 0 until that lane fills it". Lane 2 is merged, so there is no concurrent edit. Without it the exit criterion ("≥ 0.5 on co-fire") is unreachable.

Drill-down and graph

PgTargetDrillDownCollector.Queries.cs — the /* filled by lane 6 */ seam: pg_temp_spill_statements, the top five statements by differenced temp_blks_written over the window (GREATEST(x − LAG(x), 0) over the full series identity, as the bad-actor detail does), with temp_bytes_written (8 kB pages stated in bytes beside the spill fact's temp_bytes), temp_blks_read, calls, time, per-call blocks, and the normalised text + hashtext. Runs whenever the spill is on the path, beside any bad-actor detail. Lane 7's "filled by lane 6" pin still holds.

PgTargetRelationshipGraph.Query.cs — the edge PG_TEMP_SPILL → CONFIG_PG_WORK_MEM, predicate "the knob fired" (its verdict, never a size of its own). No edge into PG_BAD_ACTOR_*: the exact-key problem lane 7 recorded is unsolved; the offenders reach the reader through the drill-down and the amplifier.

What it does NOT do

No baseline-relative z on temp rate (lane 9). No PG_HIT_RATIO fact (by design, above). No edge into a bad actor. No fact when the counters are flat — each fact is emitted only when its own counter moved, the convention every family on this read keeps, and load-bearing: the plumbing e2e pins Assert.Single(facts) over a day of flat counters and lanes 2/4/7 pin total_facts over the same series (verified locally: flat series → registry fact only). No CREATE INDEX (D8). effective_cache_size never named.

Which component(s) does this affect?

• Lite
• Darling
• Lite Tests
• Darling Tests
• SQL collection scripts
• Documentation
• Full Dashboard (deprecated)
• CLI Installer (deprecated)

How was this tested?

New: Darling/Darling.Tests/PgTargetTempTests.cs (compile-verified here; first executed in CI):

• scorer: spill 0 below the floor / 0.5 at it / 1.0 at critical, threshold_lineage = 0 stamped; work_mem 0 un-stamped, 0 stamped below the floor, 0.4 at it, value not graded; deadlock 0 / 0.25 / 0.5 / 0.75 / 1.0 / 25 → 1.0, no lineage flag; TPS and hit ratio context; the deadlock pair equal to ServerHealthThresholds by value.
• through FactScorer.ScoreAll: work_mem 0 with no spill, 0 below the floor, exactly 0.5 on co-fire; the spill 0.5551 × 1.3 / × 1.6 with a temp-writing bad actor; the exact-at-floor tie resolves to ONE story PG_TEMP_SPILL → CONFIG_PG_WORK_MEM through InferenceEngine on the PG graph.
• graph: one edge out of the spill, none out of the knob, active only when the knob fired.
• advice: exact headlines and the "4 MB × 200 connections = 800 MB" arithmetic; not-known when no snapshot (never an assumed 4 MB); "the engine counted 99; 3 were captured from the log" and the not-read shape; every block has a static shape, no DDL, no effective_cache_size.
• SQL by text: reader differencing verbatim, no pg_deadlocks in the rate SQL, no block counters, $4 midpoint halves, per-fact > 0 gates, the D5 stamp, drill-down shape and cap.
• DARLING_TEST_PG-gated e2e: 4 h of climbing counters with a reset at T+2h (NULL → timestamp), a flat sibling database, 3 captured graphs, a config snapshot (work_mem default, max_connections 200), one spilling statement; asserts the clamp (239 × 120 MiB, not last-minus-first's 14,400 MiB; stats_reset_count 1, rewind 1), 24.75/hr with 99 counted / 3 captured, the TPS halves (12,019/7,200 and 12,120/7,200), the story path, the drill-down offender 777, the knob never rooting, the deadlock Critical card.

Executed locally before CI: every scorer/advice/story value above through a net10.0 harness over the built assemblies (the tie above was found this way), and the whole collector + drill-down + DarlingAnalysisService.AnalyzeAsync against a throwaway timescale/timescaledb:2.28.1-pg18 with the exact planted series — outputs match the pins to the digit. Local replay of the lineage regex, the A7 ms check, the collector FROM/JOIN census and the dialect pins over the new consts: clean.

Builds (zero new warnings): PerformanceMonitor.Analysis, Darling/PerformanceMonitor.Darling.Analysis, Darling/Darling.Tests, Lite.Tests (one pre-existing xUnit2000 in LogTailOverlapThresholdPinTests.cs, untouched), deprecated/Dashboard.Tests — all with -p:EnableWindowsTargeting=true where windows-targeted.

Checklist

• I have read the contributing guide
• My code builds with zero warnings (dotnet build -c Debug)
• I have tested my changes against at least one SQL Server version (PostgreSQL 18 store; this surface has no SQL Server twin)
• I have not introduced any hardcoded credentials or server names