A PostgreSQL operator learns how many cores their statements are burning and whether the backends are burning or waiting: pg_stat_kcache cores busy as the self-hosted CPU proxy graded only against the server's own hour-of-week routine, a burning-versus-waiting decomposition that becomes a finding on its own instrument's co-fire, and one honest unavailable fact on the measured fleet where the extension does not exist (#3691 lane 28)

[erikdarlingdata]

What does this PR do?

PARTIAL for #3691 — lane 28 (v3), the kernel family. A PostgreSQL target's analyze_server pass now has a CPU figure where no capacity percent exists: PG_CPU_BURN_CORES, the self-hosted CPU proxy — user + system + plan CPU seconds per wall second across every statement pg_stat_kcache tracks, differenced per statement with stats_since honoured and rated per collection: cores busy. It is graded ONLY by ANOMALY_PG_CPU_BURN — the window's peak AND mean cores busy against this server's own hour-of-week bucket over 30 days of pg_kernel_stats — because no core count is collected and "4.2 cores busy" is not a percentage of anything. Beside it, PG_CPU_DECOMPOSITION says whether this window's backend time was burning or waiting: kernel CPU against the wait profile the wait family already emitted, both normalised to rates, as burn_share / wait_share / io_wait_share. Checklist lines completed, inside the v3 line: "kernel-stats CPU decomposition (waiting vs burning)" and "the self-hosted CPU proxy". The residue line ("Notification on a STOCK target under a load storm cannot cross 1.5 on anomalies alone — the second confirmer is Aurora CPU") is NOT claimed: the fact that could confirm now exists; wiring it as the load family's second confirmer is a one-arm change in the shared PgTargetScorer.Anomaly.cs and is reported out of lane, as the brief preferred. Do not close.

The lie this stops telling — and the one it refuses to tell

On stock PostgreSQL there was no CPU fact at all: PG_CPU_PERCENT is Aurora's acu_utilization_percent, so a self-hosted server whose routine is 1 core busy and which today burns 5 looked exactly like its quiet Tuesday. Now the burn is a finding against the server's own normal, with the statement that burned it named by query_id.

The careful arm is the honest one. Every PostgreSQL target the fleet monitors today is Aurora, and Aurora does not ship pg_stat_kcache — the 2026-09-19 calibration (fifty clusters) had no population for this family. So the shape that runs on every real pass is "the extension is absent", and what it produces is exactly one PG_CPU_BURN_CORES fact in the unavailable shape — value 0, unavailable = 1, reason_pg_stat_kcache_absent = 1 (or reason_pg_stat_kcache_available = 1 when the server offers it and no database created it — the actionable state) — no cores, no zeros, no decomposition, no anomaly. The verdict is READ from pg_extension_availability (the daily capture, newest row within a 3-day lookback before the window's end, "installed anywhere" across the per-database rows V89's column names warn about), never inferred from the empty table; unknown (the availability collector never ran here) is silence, not absence. Executed on a throwaway store against the real analyze_server: nothing kernel-shaped roots a card, get_analysis_facts source=pg_kernel shows the one fact with its reason, and the advice says "not measurable here, and why — an absent instrument is not an idle server".

What each fact concludes, and from which table

Fact	Source	What the number is
PG_CPU_BURN_CORES	pg_kernel_stats (+ pg_extension_availability for the verdict)	Σ Δ(exec_user_time_ms + exec_system_time_ms + plan_cpu_time_ms) over the kernel source's own rated ms = cores busy. Per (database, query_id) LAG; a reset (stats_since IS DISTINCT FROM LAG(stats_since) — the NULL→timestamp first-reset trap DarlingPgDatabaseReader records — OR the counter going backwards) takes the current value WHOLE (the table's own reader's rule: the counters restarted inside the interval, so the current value is the CPU since; GREATEST(raw, 0) would read a restart as a quiet server). A delta counts only against the immediately preceding collection (prev_k = k - 1 on the window's DENSE_RANK ordinal): the collector captures the top 500 by CPU, so an identity can fall out and return with history the window never saw, and LAG alone would hand that lump to one sample. Metadata: cores_busy, cores_busy_peak, cores_busy_mean, user_share, plan_cpu_share, top_query_id + top_query_share (the CPU bad actor — a different ordering from lane 7's exec-time one, said so), kernel_observed_ms + witness_cores_busy (the same CPU over the coverage witness, the wait family's rule), reset_count. ObjectName = the top statement's database (a string seam the reconciler does not fold on).
ANOMALY_PG_CPU_BURN	the same text by alias (CpuBurnWindowSql = PgTargetFactCollector.PgTargetKernelCpuSql — one differencing, three readers) + the pg_cpu_burn_cores bucket	Peak AND mean cores busy per collection through AnomalyGate.EvaluateZScore — the #3653 pair gate (lane 17's shape): a cores-busy series is spiky by nature (one hot maintenance minute is a 4 against a routine of 1), and the mean must clear the same gate so the finding is "burning above routine for much of this window". Window read FIRST, bucket second (the WAL detector's order, for its reason: on the measured fleet the window is empty on every pass and the 30-day read must not run for nothing). baseline_ratio to the robust centre, top_query_id, mean_sigma. Folds onto PG_CPU_BURN_CORES through AnomalyToFamilies (registered by the plumbing).
PG_CPU_DECOMPOSITION	the wait facts already in the pass (pg_waits, either grade) × the kernel read	burn_share = cores_busy ÷ (cores_busy + backends_waiting); backends_waiting recovered from ANY wait fact's wait_ms ÷ share_of_wait_time (every wait fact carries its share of all waiting, CPU excluded) over the wait source's own observed ms; io_wait_share from the IO rollup. Emitted only when BOTH sides are known — a pass with no wait fact cannot say what share waiting was. wait_is_sampled says which grade the wait side rests on; the advice says "estimated from sampling" on stock. The kernel family runs LAST in CollectFactsAsync (pinned by the plumbing) precisely so it can read the wait facts rather than re-run the hardest read in v1.

The lineage shape of every bar

Every bar is unmeasured and says so — there was nothing to measure: PgCpuBurnCoresFloor = 0.5 cores (below half a core a deviation is an idle server's ripple) and PgCpuBurnCoresFallback = 4 cores (a whole small host's worth on any PostgreSQL; a young store fires only on that), both in the AnomalyThresholds PostgreSQL block with the marker (the lane-9 pin over every Pg* constant re-pinned); CpuDecompositionInformational = 0.4 (the shared advisory base — a decomposition is context), CpuDecompositionDominantShare = 0.8, CpuDecompositionCoFireBoost = 0.5 (0.4 × 1.5 = 0.6, past the incident line, well short of the notify line). The calibrating read is named once: the per-server burn_share distribution over pg_kernel_stats × the wait source, four-hour windows, on a stock population. No SQL Server CPU constant is reused by value (pinned against PgCpuFloorPct / PgCpuFallbackPct / the capacity ladder — those are PERCENT, this is cores). The decomposition and the anomaly stamp threshold_lineage = 0. PG_CPU_BURN_CORES stamps 1 — a deliberate divergence from the brief's "all 0", following PG_WAL_VOLUME_SHIFT's rule for the identical shape (a context fact graded only through its anomaly): no bar was chosen for the fact, so there is no unmeasured number for the flag to disclose; the anomaly's fact discloses its own. Two facts in one shape with two flags would be the inconsistency; a one-line flip if the coordinator rules otherwise.

The co-fire gating, and why the graph starts where it does

PG_CPU_BURN_CORES scores 0 always (PgTargetScorer.Kernel.cs) — the PG_WAL_VOLUME_SHIFT rule — so it can neither root a story nor be walked to (InferenceEngine's lookup excludes base 0); the instrument is the anomaly, a deviation-scored card graded by the shared ramp. A consequence stated plainly: with AnomalyAmplifiers(ANOMALY_PG_CPU_BURN) empty in the shared Anomaly.cs, the burn anomaly is a solo card that cannot reach the extremity escape (needs two co-fires) and tops out under the 1.49 cap; the fold onto PG_CPU_BURN_CORES is reachable only if a later lane grades the proxy. Both are the out-of-lane items below.

PG_CPU_DECOMPOSITION roots at 0.4 (informational; nothing walks to it) and crosses to 0.6 on exactly one co-fire: compute-bound (burn_share ≥ 0.8 AND ANOMALY_PG_CPU_BURN fired — the anomaly source is in ScoreAll's lookup, the write family's trigger-amplifier precedent) or wait-bound (wait_share ≥ 0.8 AND a PG_WAIT_* fact is itself a finding). Exclusive by arithmetic, so 0.6 and no higher. Edges leave the decomposition (PgTargetRelationshipGraph.Kernel.cs, category cpu_burn): compute-bound → PgTargetFactKeys.BadActorFamily (the FIRST edge in the tree to name the alias; resolves to the highest-severity PG_BAD_ACTOR_*, dropped when none is emitted, never thrown); wait-bound → each of the eight v1 wait keys whose fact fired, standouts before rollups (a rollup whose standout fired scores 0 and is never chosen over it). No edge from the proxy (dead by construction — the write chain removed its PG_WAL_VOLUME_SHIFT edge for the same reason) and none from the anomaly (no anomaly has one; the fold is the relationship). The plumbing's graph-header claim that PG_CPU_PERCENT reaches the bad actor in Saturation.cs was false (no such edge exists) and is rewritten.

Advice (OtterTune doctrine)

Value-stated from the facts: "CPU burn: 5.05 cores busy across the window, 5.2 cores at peak … 85 % of the execution CPU in user time … Statement 7001 in app burned 75.9 % of it — the CPU bad actor, ranked by kernel time, which is not the same ordering as the elapsed-time top statements … not a percentage of anything … never a host CPU percent." The anomaly: the shared deviation sentence ("CPU burn spiked to 5.2 cores busy — 8σ above its baseline for this time of week"), "That is 5× the 1 core this server routinely keeps busy at this hour of the week", the statement, and "How many cores this server has is not known from this reading — 5.2 cores busy may be a fraction of a large host or all of a small one". The decomposition says which side dominates, at which grade ("estimated from sampling" / "the engine's measured wait time"), and where to look next. Levers with counter-objectives: the statement first (get_pg_kernel_stats ranks by burn, get_pg_top_queries by elapsed — different on purpose); max_parallel_workers_per_gather (more cores for one query, fewer for the rest); jit (compile CPU paid back only on long analytic statements); a high plan-CPU share → prepared statements / plan_cache_mode at the cost of the generic plan. No CREATE INDEX (D8 — pinned across every block); fsync / synchronous_commit named nowhere (D6 — pinned). The static block for the anomaly is built on first use, not as a static readonly initializer (the cross-partial hedge-aliasing trap the write family records). next_tools: get_pg_kernel_stats, get_pg_top_queries, get_pg_extensions / get_pg_wait_stats — all get_pg_*.

Retention

pg_kernel_stats joins DarlingRetentionHorizons.BaselineServingRawCollectors by collector name (its 30-day schedule default was, again, the only thing covering the baseline window); BaselineSupplyTests derives the set from the arm's text and its two counts moved (9 → 10 sources; 11 → 12 members, also in PgTargetBetweenWavesV2Tests).

What it does NOT do

• Does not wire ANOMALY_PG_CPU_BURN as the load family's second confirmer, nor give it the load family's co-fires — both one-arm changes in the shared PgTargetScorer.Anomaly.cs (reported below for the between-waves batch). Until then the burn anomaly is a solo card under the 1.49 cap, and the residue line stays open.
• Does not grade the proxy against a core count — none is collected. Does not read exec_read_bytes / major_faults (a later slice, if any). No new key, source, schema, rung or knob.
• The unavailable fact's verdict is as fresh as the daily availability capture (3-day lookback); an extension created this morning reads "not installed" until tomorrow's run — the advice says the rows arrive on the collector's next run.
• The re-entry rule makes the sum a FLOOR on a churning top-500 (stated in the SQL's doc and on the fact through reset_count / rated_samples).

Rebased over the siblings that landed under it

Rebased onto origin/dev @ 805ec01 after the first head went CONFLICTING against R5 61c7596 (V136, DailySummaryHorizon floor set), lane 27 4991595 (the plans family) and lane 29 bea6b41 / 26ec0da (buffer drill-down). Every conflict was a roster / count / constant-block collision and was resolved to the UNION: the floored set carries lane 27's pg_statement_stats AND pg_kernel_stats (BaselineSupplyTests 10 → 11 sources, PgTargetBetweenWavesV2Tests 12 → 13 members); PgTargetAnomalyTests serves 11 metrics, fences eleven filled detectors and — with both v3 stubs now filled — replaces the exemption loop with a positive pin that no PgTargetAnomalyDetector.*.cs is a Task.CompletedTask stub; PgTargetV3PlumbingTests' inert loops for the anomaly and the metric would both have been empty and are replaced by the kernel's filled-shape assertions beside lane 27's; the // lane 28 block in AnomalyThresholds follows lane 27's; the lane-28 lines in PgTargetSharedSwitchRoutingTests sit after lane 27's anomaly pin. Re-executed on the rebased tree: 944 / 944 green, 0 skipped (every PgTarget*Tests file incl. lanes 27 and 29's, plus the census set), same rig, torn down.

Which component(s) does this affect?

• Lite
• Darling
• Lite Tests
• Darling Tests
• SQL collection scripts
• Documentation
• Full Dashboard (deprecated)
• CLI Installer (deprecated)

How was this tested?

New: Darling/Darling.Tests/PgTargetKernelTests.cs (11 tests, 2 gated live, [Collection("live-postgres")] + LiveStoreCleanup). The two reads' shape (per-identity LAG, reset by stamp OR backwards counter taken whole, DENSE_RANK re-entry rule, no JOIN, server_id = $1, no now(), the detector's text by alias, availability before rows, the fence); the baseline arm (served, the one scaffold, half-open, ::DOUBLE PRECISION, no own EXTRACT, retention floored); the bars (values, unmeasured markers, no SQL Server CPU value reused); the arithmetic executed on arranged rows through the EmitKernelFacts seam (cores busy over the source's own time, shares, top statement, ObjectName, witness figure, lineage 1, base 0, no amplifier; the decomposition from planted wait facts with both sides normalised, sampled flag, lineage 0); every co-fire path through FactScorer.ScoreAll (compute-bound + anomaly → 0.6; anomaly alone → 0.4; 0.79 → 0.4; wait-bound + fired wait → 0.6; wait under its bar → 0.4; the proxy never lifted); the unavailable shape (one reason, no zeros, advice says which); the edges (nine declared; the alias resolves to the statement, is dropped when none, wait-bound lands on the fired standout not the quiet rollup, mixed → nothing, none from proxy or anomaly); the advice pins across all three keys × static and composed. Live, executed here: (a) the Aurora honesty arm above; (b) a stock-stamped server with installed, 31 days of one-minute pg_kernel_stats for two statements at ~1 core (5 % jitter) then 4 hours at 5 cores with 7001 at ~77 %, plus a light measured pg_wait_stats profile — bucket trusted (median ≈ 1), collector: 5 cores / 240 rated / 7001 / user share 0.85 / compute-bound, detector: the anomaly at 5 cores, lineage 0, ratio ≈ 5, pair gate passed; analyze_server: the anomaly roots with the prose above, PG_CPU_DECOMPOSITION roots at exactly 0.6 saying compute-bound, nothing roots on the proxy, every next_tools entry is get_pg_*; get_analysis_facts source=pg_kernel: the proxy at base 0 with its cores and lineage 1, the split with burn_share ≥ 0.8 and lineage 0.

Edited pins, each deliberately: PgTargetAnomalyTests (served metrics 9 → 10 with (pg_cpu_burn_cores, pg_kernel_stats); the kernel metric left the inert loop; CpuBurnWindowSql in the detector SQL census; the Kernel detector moved from the exemption list to the filled-and-fenced list with its pair-gate pin; the Pg* constant roster gained the two); PgTargetSharedSwitchRoutingTests (the three lane-28 lines Null → NotNull, the anomaly never the SQL Server composer); PgTargetV3PlumbingTests (the kernel rows left the inert loop, as the comment now says each lane's do); BaselineSupplyTests / PgTargetBetweenWavesV2Tests (the counts); PgTargetFactCollectorTests (one pin line: pg_extension_availability, the first analysis read of it). PgTargetFactCollectorTests.AllSql_… itself is unchanged.

Executed, not just compiled: a net10.0 harness (AssemblyName=Darling.Tests, inside the worktree under obj/, deleted before commit) against a throwaway timescale/timescaledb:2.28.1-pg18 on Docker context desktop-linux (role and database darling, migrated by PgMigrations.MigrateAsync), torn down after: 908 / 908 green, 0 skipped — every PgTarget*Tests file, PgTargetThresholdLineageTests, PgTargetMeasuredLineageTests, BaselineSupplyTests, LocalClockBucketKeyTests, BaselineTierJoinShapeTests, StoreSqlClockDisciplineTests, FactSourceRegistryTests, CollectionCaveats*Tests, FleetIdentifierScrubTests, DocCommentHygieneTests, RepoFileAdoptionTests, CommentFilterAdoptionTests, LiveCleanupConversionRatchetTests, LivePostgresCollectionHygieneTests, AsOfWindowAnchorTests, AnalysisPass*Tests, AnalysisShutdownResidueTests, DarlingAnalysisServiceEngineRoutingTests, RobustBaselineTests, BaselineTimeoutIsNamedTests, including every DARLING_TEST_PG e2e in that set. One shipping defect the execution caught that compile-only would have shipped: the first draft's re-entry rule was a JOIN from the per-identity window CTE back to a distinct-collection-times CTE; on 89k fresh, unanalysed rows the planner nested-looped it and the baseline read ran past its 60-second timeout — samples=0, the anomaly silent. Rewritten as a DENSE_RANK ordinal on the same rows: 0.27 s on the same data. Pinned (DoesNotContain("JOIN") on both texts). First Windows execution is CI.

Builds, zero warnings (-p:EnableWindowsTargeting=true on the test projects): PerformanceMonitor.Analysis, PerformanceMonitor.Darling.Analysis, Darling.Tests, Lite.Tests, deprecated/Dashboard.Tests. No production store or fleet server was read.

Checklist

• I have read the contributing guide
• My code builds with zero warnings (dotnet build -c Debug)
• I have tested my changes against at least one SQL Server version (PostgreSQL-target lane; executed against PostgreSQL 18 + TimescaleDB 2.28.1 — every new arm is behind a pg_ source or PG_ / ANOMALY_PG_ key no SQL Server fact carries, and the routing pins that proved v1's byte-identity pass unchanged)
• I have not introduced any hardcoded credentials or server names

Partial for #3691 (lane 28, the kernel family). CHANGELOG deliberately untouched — the coordinator splices this wave's entries.