Skip to content

v3 plumbing for the PostgreSQL-target engine: the plan, kernel and memory families exist as stubs behind every shared switch, so a PostgreSQL operator sees nothing new yet and the content lanes never touch a shared file (partial #3691) - #3792

Merged
erikdarlingdata merged 1 commit into
devfrom
feat/3691-v3-plumbing
Sep 20, 2026

Conversation

@erikdarlingdata

Copy link
Copy Markdown
Owner

What does this PR do?

Headline mechanism. The v3 plumbing keystone for the PostgreSQL-target analysis engine (#3691) — what #3665 did for v1, #3715 for v2 and #3737 for the wave-3 blocking family, done again for the plan, kernel and memory families: the v3 vocabulary is declared once (PgTargetSources / PgTargetFactKeys / MetricNames), every new family exists as a stub in its own partial file with a /* filled by lane N */ marker, every shared switch grew exactly one delegating arm per family, both anomaly composers are root-declared with their ComposeAnomaly case (the #3737 hook shape, so no content lane edits the anomaly partial), and the censuses that pin the shared surface are re-pinned deliberately. Lanes 27 (plans), 28 (kernel), 32 (memory) fan out from this merge and never touch a shared file again; lanes 29 (buffer drill-down) and 30 (Seq-Scan advisory) are content lanes on existing / plan files and needed no stub beyond the plan family's. PARTIAL for #3691 — v3 plumbing; no checklist line is this PR's. Do not close.

No content: no threshold, no advice prose, no graph edge, no SQL (the stubs' AllSql contribution is empty), no rung (StorageVersion stays where dev has it — V136 is lane R5's), no AnomalyThresholds constant. No operator-visible change: the analyze_server gated e2e returns the same payload (executed, below).

Every declaration, stub and switch arm — with the lane that fills it

Sources (PgTargetSources, each ending in Source for the registry sweep): PlansSource = "pg_plans" (27), KernelSource = "pg_kernel" (28), MemorySource = "pg_memory" (32). All 15 → 18 (ordinal order kept); FactScorer.KnownSources 30 → 33; both SKUs' get_analysis_facts source description spells the new list (the one Lite edit — the census-pin boundary extension every plumbing PR takes).

Keys (PgTargetFactKeys, doc-commented with design § and table):

  • plans (§6; pg_plan_capture × pg_statement_stats, pg_column_stats, pg_predicate_stats): PlanRegression = "PG_PLAN_REGRESSION", ParameterSensitivity = "PG_PARAMETER_SENSITIVITY", SeqScanAdvisory = "PG_SEQ_SCAN_ADVISORY" (lane 30 — EVIDENCE only, never DDL, D8; the doc says the maintainer's decision gates the lane), AnomalyPlanRegression = "ANOMALY_PG_PLAN_REGRESSION".
  • kernel (pg_kernel_stats, pg_stat_kcache; availability from pg_extension_availability): CpuBurnCores = "PG_CPU_BURN_CORES" (the self-hosted CPU proxy), CpuDecomposition = "PG_CPU_DECOMPOSITION", AnomalyCpuBurn = "ANOMALY_PG_CPU_BURN".
  • memory (§4b; pg_server_config × the V136 host-memory columns on pg_cpu_utilization's row): ConfigMemoryOvercommit = "CONFIG_PG_MEMORY_OVERCOMMIT", HostMemoryPressure = "PG_HOST_MEMORY_PRESSURE".
  • NOT declared, deliberately: a max_wal_size-vs-disk composition key — disk free is not collected, and a key nothing can measure is a lie waiting for a lane. Lane 29 declares nothing: it is a drill-down on PG_BUFFER_CACHE_PRESSURE over pg_buffer_usage, lane 16's deadlock shape.

AnomalyToFamilies gains AnomalyPlanRegression → PlanRegression, AnomalyCpuBurn → CpuBurnCores. PgTargetScorer.IsDeviationScoredAnomalyKey registers both by shape (z-score on one series). ConfigAdvisoryRoots gains ConfigMemoryOvercommit — a CONVENTION reading of five knobs against the host roots at the 0.4 advisory base and reaches ≥ 0.5 only when a workload co-fire (PG_HOST_MEMORY_PRESSURE or PG_TEMP_SPILL) lifts it (D5); the content lane decides which and pins it, this PR only routes. HostMemoryPressure is measured, not config, and is NOT a root (pinned).

Baseline metric names (MetricNames): PgStatementMeanMs = "pg_statement_mean_ms" (27), PgCpuBurnCores = "pg_cpu_burn_cores" (28). The plan name's doc records the decision lane 27 owns FIRST: PgBaselineProvider.GetBaselineAsync keys one series on (server, metric) with no per-statement dimension, so the arm is either the server-wide mean (blunt, honest) or the seam grows a key dimension — a shared-file item to report, never made inside the lane. The stub takes no side.

Stub partials, each with the exact marker the content briefs quote:

File Marker Stub method
PerformanceMonitor.Analysis/PgTargetScorer.Plans.cs /* filled by lane 27 */ ScorePlanFact → 0.0, PlanAmplifiers → []
…/PgTargetScorer.Kernel.cs /* filled by lane 28 */ ScoreKernelFact → 0.0, KernelAmplifiers → []
…/PgTargetScorer.Memory.cs /* filled by lane 32 */ ScoreMemoryFact → 0.0, MemoryAmplifiers → []
…/PgTargetAdvice.Plans.cs /* filled by lane 27 (…) and lane 30 (…) */ + /* filled by lane 27 — the ANOMALY_PG_PLAN_REGRESSION arm … ComposePlan → null, ComposePlanRegressionAnomaly → null
…/PgTargetAdvice.Kernel.cs /* filled by lane 28 */ + /* filled by lane 28 — the ANOMALY_PG_CPU_BURN arm … ComposeKernel → null, ComposeCpuBurnAnomaly → null
…/PgTargetAdvice.Memory.cs /* filled by lane 32 */ ComposeMemory → null
…/PgTargetRelationshipGraph.Plans.cs / .Kernel.cs / .HostMemory.cs /* filled by lane 27 */ / 28 / 32 BuildPlanEdges / BuildKernelEdges / BuildHostMemoryEdges — empty
Darling/…/PgTargetFactCollector.Plans.cs / .Kernel.cs / .Memory.cs /* filled by lane 27 (lane 30 adds …) / /* filled by lane 28 / /* filled by lane 32 CollectPlanFactsAsync / CollectKernelFactsAsync / CollectMemoryFactsAsync → Task.CompletedTask
Darling/…/PgTargetAnomalyDetector.Plans.cs / .Kernel.cs /* filled by lane 27 / 28 DetectPlanRegressionAnomalies / DetectCpuBurnAnomalies → Task.CompletedTask
Darling/…/PgTargetBaselineProvider.Plans.cs / .Kernel.cs /* filled by lane 27 / 28 StatementMeanMsBaselineQuery / CpuBurnCoresBaselineQuery → null

The memory family's graph partial is HostMemory.cs because PgTargetRelationshipGraph.Memory.cs is lane 2's v1 buffer chain (BuildMemoryEdges); one source node's edges live in one place, and the root comment says so. The collector method is CollectPlanFactsAsync in Plans.cs (the CollectQueryFactsAsync / Queries.cs shape) — the method→file maps in PgTargetFactCollectorTests and CollectionCaveatsTests each gained the line. Every baseline stub's doc states the Q6 contract the filled arm must follow (clean(collection_time, v) + the ONE RobustTierScaffold, bound through the root's $4..$6 after $3, never its own EXTRACT); every detector stub's doc names the #3653 pair gate; every collector stub's doc names the timeout / token / $N / clock / degrade / observed-time rules.

Shared-switch arms (one per family, nothing else moved):

  • PgTargetScorer.ScoreBase: PlansSource → ScorePlanFact, KernelSource → ScoreKernelFact, MemorySource → ScoreMemoryFact. Amplifiers: the three plan keys → PlanAmplifiers; the two kernel keys → KernelAmplifiers; ConfigMemoryOvercommit or HostMemoryPressure → MemoryAmplifiers, named AHEAD of the CONFIG_PG_ prefix arm because the check is pg_memory-sourced (the ConfigSharedBuffers-in-BufferAmplifiers precedent; pinned by ordinal position in source).
  • PgTargetAdvice.Compose: the same three groups → ComposePlan / ComposeKernel / ComposeMemory, the memory arm again ahead of the prefix arm (which would compose null for it forever — the ConfigAutovacuumDisabled precedent). ComposeAnomaly: case AnomalyPlanRegression → ComposePlanRegressionAnomaly, case AnomalyCpuBurn → ComposeCpuBurnAnomaly, both partials declared in the root file.
  • PgTargetRelationshipGraph ctor: BuildPlanEdges(); BuildKernelEdges(); BuildHostMemoryEdges(); after the blocking chain.
  • PgTargetFactCollector.CollectFactsAsync: the three stubs LAST, after CollectBlockingFactsAsync — the plan facts name lane 7's bad actors, the kernel decomposition reads the wait facts, the memory arithmetic reads the config knob facts by key (pinned by position).
  • PgTargetAnomalyDetector.DetectAnomaliesAsync: the two stubs after the sampled wait profile, behind the same gate. PgTargetBaselineProvider.GetPgTargetBaselineQuery: two arms to the stubs (null = the default's answer).
  • PgTargetToolRecommendations.ByFactKey: a row for every new key — every tool verified registered by rg over the Mcp folder: get_pg_plans (there is no get_pg_plan_captures), get_pg_top_queries, get_pg_query_duration_trend, get_pg_column_stats, get_pg_predicate_stats, get_pg_kernel_stats, get_pg_extensions, get_pg_wait_stats, get_pg_server_config, get_pg_buffer_usage, get_pg_session_states, get_pg_cpu_utilization. The host-memory columns ride pg_cpu_utilization's row (R5's V136 shape), so the pressure fact's first read is the capacity read; no memory tool exists or is needed.

Censuses re-pinned (each edited deliberately, none deleted)

  • PgTargetFactCollectorTests: collect surface 16 → 19 names, stub files 14 → 17, the three markers pinned by exact string, CollectPlanFactsAsync → Plans.cs in the method→file map. FROM/JOIN whitelist: pg_plan_capture, pg_statement_stats, pg_column_stats, pg_predicate_stats, pg_kernel_stats, pg_buffer_usage, pg_cpu_utilization are CollectorCatalog targets already — admitted with no edit, each now pinned BY NAME with its lane. pg_database_size is NOT pinned (it arrives with R5's catalog row; the first lane to read it adds the pin).
  • PgTargetSharedSwitchRoutingTests: sources 15 → 18; AnomalyToFamilies two entries; ConfigAdvisoryRoots contains the overcommit key and not the pressure key; the advice delegation census gains nine Null lines (null IS the delegation — each lane moves its own lines to NotNull); the routing census EveryDeclaredPgKey_IsRoutedByEveryRootDispatcher… passes with zero exclusions added (every new key is routed by name; the floors ≥ 25 / 12 / 9 hold).
  • PgTargetAnomalyTests: anomaly keys 10 → 12 (both deviation); the inert-null loop gains the two metric names; the provider-arm text pins gain two lines; the detector-order check gains two calls; the stub-marker loop gains four files; the fence test names the two v3 stubs as exemptions by exact stub text (the The PostgreSQL engine absorbs what the v2 lanes reported back: the I/O and WAL baseline tables are floored, the I/O baseline restores hour-of-week at a quarter-hour grain, the WAL co-fire reads the anomaly and not a base-0 fact, a Lock wait reaches its idle holder, and the blocking family exists as stubs for lane 17 (partial #3691, between waves) #3737 shape).
  • FactSourceRegistryTests: 30 → 33. CollectionCaveatsTests / PgTargetCollectionCaveatsLiveTests: the family-count denominator 16 → 19 (a stub is a family READ that returns nothing; the count is reads, not facts — it renders only inside a collection caveat, so a clean pass's payload is byte-identical), plus the ["plan"] = "plans" label map; the root's "1 of 16" example → "1 of 19".
  • New Darling/Darling.Tests/PgTargetV3PlumbingTests.cs (3 tests): the vocabulary + folds + advisory root + no disk key; inert through every shared entry point (base 0, no amplifier, null advice static and via FactAdvice, no edge, null baseline in both providers, a tool row) with the memory arm's position ahead of the prefix arms pinned as source; the sixteen stub files' markers, no aurora_stat_, no CREATE INDEX in any string literal, Memory.cs untouched, both anomaly hooks root-declared, the collector emission order.
  • PgTargetThresholdLineageTests: zero bars added. AuroraOnlySqlIsGatedTests: no new file mentions aurora_stat_ (grepped + pinned). DocCommentHygiene, RepoFileAdoptionTests (single-line anchors, ReadRepoFile not Lf), CommentFilterAdoptionTests, StoreSqlClockDisciplineTests, BaselineSupplyTests (null arms contribute nothing), LocalClockBucketKeyTests: green, no edit.

What it does NOT do

No PostgreSQL SQL, no threshold, no advice text, no graph edge, no baseline CTE, no AnomalyThresholds constant, no rung, no DailySummaryHorizon.BaselineServingRawCollectors floor (the stubs' arms are null, so BaselineSupplyTests derives nothing to floor — pg_statement_stats / pg_kernel_stats join the set the day lanes 27 / 28 make their arm non-null, and each stub's doc says so). No MaintenanceShapeShift stub. A SQL Server target's pass is byte-identical by construction: every new arm is behind a pg_ source or a PG_ / CONFIG_PG_ / ANOMALY_PG_ key no SQL Server fact carries, and the routing pins that proved v1's byte-identity pass unchanged.

Test plan

Builds, zero warnings: PerformanceMonitor.Analysis, PerformanceMonitor.Darling.Service (pulls Darling.Analysis + Storage), Darling.Tests, Lite.Tests, deprecated/Dashboard.Tests (-p:EnableWindowsTargeting=true on the test projects).

EXECUTED here through a net10.0 harness (AssemblyName=Darling.Tests, the PgTarget census set — every PgTarget*.cs plus FactSourceRegistryTests, CollectionCaveats*Tests, BaselineSupplyTests, LocalClockBucketKeyTests, BaselineTierJoinShapeTests, StoreSqlClockDisciplineTests, DocCommentHygieneTests, RepoFileAdoptionTests, CommentFilterAdoptionTests) against a throwaway timescale/timescaledb:2.28.1-pg18 on Docker context desktop-linux, migrated by the fixture, torn down after: 744 / 744 green, 0 skipped, on the rebased tree (origin/dev @ 69a3a78, after lane 26's #3791). That includes every DARLING_TEST_PG-gated e2e in the set — the exit criterion: PgTargetFactCollectorTests.APostgresTargetWithADayOfHistory… (analyze_server → the same payload; the stubs emit nothing) and PgTargetCollectionCaveatsLiveTests (a clean pass carries no caveat; a blind vacuum family says 1 of 19). DocCommentHygieneTests ran green with the harness under obj/ (gitignored, deleted before commit). First execution of the Lite literal's pins is CI's Windows job.

Which component(s) does this affect?

  • Lite (one string literal: the get_analysis_facts source description, a census pin)
  • Darling
  • Lite Tests
  • Darling Tests
  • SQL collection scripts
  • Documentation
  • Full Dashboard (deprecated)
  • CLI Installer (deprecated)

How was this tested?

Above. No production store or fleet server was read; the rig was a local container, torn down.

Checklist

  • I have read the contributing guide
  • My code builds with zero warnings (dotnet build -c Debug)
  • I have tested my changes against at least one SQL Server version (not applicable — the SQL Server pass is unreachable by every new arm; the routing pins that proved v1's byte-identity pass unchanged)
  • I have not introduced any hardcoded credentials or server names

Partial for #3691 (v3 plumbing; lanes 27–32 follow). CHANGELOG deliberately untouched — the coordinator splices this wave's entries.

…mory families exist as vocabulary and inert stubs, every shared switch has its arm, and the censuses are re-pinned (partial #3691)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant