Stock PostgreSQL's sampled waits read per second the sampler was watching (V133 sampled_ms — ten times what the interval arithmetic said on the service sampler), get their own baseline and ANOMALY_PG_SAMPLED_WAIT_PROFILE detector folding onto the dominant sampled wait, and the exact source wins when both wait tables write one window (partial #3691, lane 24)

[erikdarlingdata]

Stock PostgreSQL's sampled waits now read per second the sampler was WATCHING, have their own baseline and detector, and say what happens when both wait sources write one window

PARTIAL for #3691 — lane 24. Completes these checklist lines VERBATIM:

• (rung half DONE …) Stock PostgreSQL's sampled waits get no anomaly baseline — the consumer half: the pg_wait_sampling rate divides by V133's sampled_ms, and the sampled profile gets its own metric (pg_sampled_wait_ms_per_sec), its own key (ANOMALY_PG_SAMPLED_WAIT_PROFILE) and its own bar.
• "PgSampledWaitProfileFallbackMsPerSec deliberately undeclared" — declared now, by the detector that reads it, on its own lineage.

Does not close the issue.

The lie the product told

Two instruments write pg_wait_sampling in the same shape — sample_count cumulative, profile_period_ms beside it. The pg_wait_sampling extension watches the whole interval in-engine; the #3604 service-side sampler watches thirty one-second snapshots of every 300 s cycle. Lane 5's stock read (Δsamples × period over the interval between collections) is honest for the first and a ~10× understatement for the second, and until V133 (#3716) nothing in the row said which arm wrote it — the instrument token lived in collector_state, outside the analysis whitelist. So a stock server whose backends were seen waiting one full backend-equivalent continuously read as a tenth of one, under every bar, and the fleet's stock population had no notion of its own normal for waiting at all: v1 shipped the sampled estimate with no baseline and no anomaly, deliberately, because pooling a per-backend-sample count quantised at the period with the engine's measured microsecond sum would have been the unit error #3689 §5 refused.

What it does now, and from which table

The honest rate (PgTargetFactCollector.Waits.cs). Per countable collection the observed time is coalesce(sampled_ms / 1000, LAG interval) — the milliseconds THAT collection's sampler actually watched when the row carries it (30,000 on the sampler arm), the wall interval when it does not. The fraction on every sampled PG_WAIT_* fact is now ms of sampled waiting per second watched. A pre-V133 row is NULL and came from EITHER arm — indistinguishable from the extension — so NULL is read as the whole interval (exactly lane 5's arithmetic) and never guessed at 30 s; a window that straddles the install mixes the two honestly per collection, both being milliseconds observed. New metadata on every sampled fact: wait_source_interval_ms (the wall time between the same collections, so the duty cycle is stated, not inferred) and sampled_ms_known (1 only when every countable collection disclosed its watched time; 0 when any was read the old way). wait_source_observed_ms is now the watched time; witness_fraction (the same wait over the pass's 4 h) keeps the old figure beside it. The live pin asserts the exact 10×: the same 48 cycles read 0.15 over the interval and 1.5 over the watched time.

Both sources in one window: the exact one wins, and says so. The Aurora read's header now counts pg_wait_sampling's collections over the same window (one index probe). When both wrote — a re-platformed target, a sampler left running beside the engine's counters — the sampled facts are not emitted and every exact fact carries sampled_suppressed_by_exact = 1; the sampled detector reads pg_wait_stats' count on its own row and sits out. Lane 5's read-then-fallback read only one source and could not see the case (its body said so); the choice is now explicit on the fact and pinned live on a server planted with both.

The baseline (PgTargetBaselineProvider.WaitsSampled.cs). pg_sampled_wait_ms_per_sec: per collection, the reader's reset-aware difference of every (event_type, event, query_id) series (a count that went backwards is taken whole, never GREATEST's zero; a series' first in-window sighting contributes nothing) × period, CPU/Running excluded, over coalesce(sampled_ms / 1000, interval). Ends in clean(collection_time, v) + the ONE RobustTierScaffold, so it inherited lane 20's local-clock keying (#3757) with zero edits — LocalClockBucketKeyTests executed green over it. Never pg_wait_ms_per_sec: same unit, different instrument, separate buckets. pg_wait_sampling joins BaselineServingRawCollectors (D10; BaselineSupplyTests derives the set from the arm's SQL and moved 8 → 9).

The detector (PgTargetAnomalyDetector.WaitsSampled.cs). Lane 9's Aurora wait-profile detector with the instrument swapped and the #3653 peak-AND-mean gate (#3724) the ratio family did not yet have: robust bucket → modified z at the heavy-tail cutoff on the peak AND the window mean, plus the magnitude bar on the peak; classical → the 3× ratio on both, plus the bar; untrustworthy → the bar on both, is_new, no sentinel. One hot five-minute collection is not a profile shift, and a 24-hour anchored pass does not mechanically fire more than the 4-hour one. Top six contrib_Type:event contributors by estimated ms; is_sampled, wait_source_observed_ms, wait_source_interval_ms, sampled_ms_known, mean_ms_per_sec, mean_modified_z, mean_ratio on the fact; threshold_lineage = 0. Sits out on Aurora (no sampler rows), on both-sources, on no rated collection, on an empty bucket.

The lineage shape of every bar

Bar	Shape
PgSampledWaitProfileFallbackMsPerSec = 500 (floor and fallback, one number on both paths)	unmeasured — chosen, not measured; the 2026-09-19 calibration read Aurora's exact deltas, and the stock pg_wait_sampling rate at the sampled_ms denominator has no distribution yet (every pre-V133 row is NULL). Equal in value to the Aurora bar and deliberately not an alias (pinned: the declaration is a literal), so the two instruments calibrate apart. Lane 9 left it undeclared while nothing read it; the detector that reads it is this one.
Heavy-tail cutoff (5.0), PgRatioAnomalyThreshold (3), the modified-z span	by reference — the Aurora profile's, unchanged
Lane 5's wait bars 0.20 / 1.0 / IO 0.40 / 2.0 (measured on Aurora's exact deltas)	applied to the sampled rate as before; the constants' comments now state: the sampled rate on stock is the same unit from a different instrument — unmeasured for stock — and every sampled fact keeps threshold_lineage = 0, its advice says "estimated from sampling". PgTargetMeasuredLineageTests and PgTargetThresholdLineageTests executed green.

A first-month reading to expect, stated in the detector's doc: on a fleet that has just installed V133 the window is honest and the 30-day bucket is mostly blind (pre-V133 NULL rows read ~10× low on the sampler arm), so the first passes can fire on the arithmetic change alone. The fact says sampled_ms_known; the advice says the baseline was built the same way.

The co-fire gating and the fold

The sampled profile takes exactly the Aurora profile's five amplifiers through PgTargetFactKeys.IsWaitProfileAnomaly — the three load siblings, the capacity-measured CPU confirmer (inert on stock: no PG_CPU_PERCENT fact) and "a named PG_WAIT_* standout crossed its own threshold" (which on stock reads the sampled facts). Same extremity escape on the same statistic (IsExtremeWaitProfileAnomaly); same ScoreRatioAnomaly ramps. It folds per story onto its dominant contributor's own card through WaitProfileFamilies (standout first, rollup second), exactly as the Aurora one does since #3715 — not a static AnomalyToFamilies entry. On stock, one confirmer: the live run scores the folded anomaly at exactly 1.3 — released from the cap by the escape, under the 1.5 notify floor — the stock ceiling PREAMBLE-3691 states (a v3 item, not this lane's).

Advice (PgTargetAdvice.Wait.cs)

The sampled wait facts' evidence sentence now reads "Over 24 min the sampler was watching (48 collection intervals spanning 4 h of wall clock — the sampler watches part of each cycle and the fraction is per second watched)", and when sampled_ms_known = 0: "Some or all of these collections did not record how long the sampler watched (rows written before V133, or the pg_wait_sampling extension …) so they are read over their whole interval; on the service-side sampler that reads about a tenth of the true rate." The witness-divergence sentence compares the WALL interval to the witness on a sampled fact (the watched time is a fraction of it by design and has been stated). ComposeSampledWaitAnomaly: the Aurora composer's shape in the sampled grade's words — peak and mean ms per second watched, the multiple or the robust sigmas, the leading contributors, the duty cycle, the pre-V133 caveat, or the first-occurrence rendering with no sigma. Says "estimated from sampling"; never "the engine measured", never "spent" (pinned).

Boundary extensions, each one line, each stated

The brief granted the root one-liners, MetricNames, AnomalyThresholds, PgTargetFactKeys, DailySummaryHorizon and named PgTargetScorer.Anomaly.cs's IsExtremeWaitProfileAnomaly key check ("one or; state it"). Routing the key through EVERY shared switch needed the same predicate in four sibling sites, so I put PgTargetFactKeys.IsWaitProfileAnomaly(key) in the granted keys file and pointed each site at it rather than adding literals:

• PgTargetScorer.Anomaly.cs — IsPgRatioAnomalyKey (+1 or), IsExtremeWaitProfileAnomaly (predicate), ScoreRatioAnomaly's modified-z arm (predicate), AnomalyAmplifiers's profile arm (predicate). Without the last two the key would have graded on the ratio arm only and had no amplifiers.
• AnomalyIncidentReconciler.cs:316 — the AnomalyWaitProfile equality became the predicate. Without it the story stays solo beside the very sampled wait card it is about, which the brief's exit criterion ("folded onto the dominant wait") forbids.
• PgTargetAdvice.Anomaly.cs — one case arm delegating to the wait file (the routing census PgTargetSharedSwitchRoutingTests requires a case per ANOMALY_PG_* key; lanes 12/15/17 each added theirs).
• Mcp/PgTargetToolRecommendations.cs — one row (PgTargetMcpSurfaceTests requires a next_tools row per declared key).
• PgTargetBetweenWavesV2Tests.cs:49 — the floored-set count 10 → 11 (the pin the floor moves).

The collector's EmitWaitFacts seam changed shape (sampled gains IntervalSec / SampledMsKnown; extra is a list — the exact arm now stamps two keys); its callers are the collector and PgTargetWaitTests, both updated.

What it does NOT do

• Does not close PostgreSQL-target analysis engine: v2/v3 slices and the v1 lanes' residue (from #3542) #3691. Does not touch PgWaitSamplingCollector, the reader, the store or any migration (V133 is pg_database_stats stores numbackends and pg_wait_sampling stores sampled_ms (Darling V133), so a PostgreSQL operator's saturation fraction has a numerator and the sampler arm's wait rate has its duty-cycle denominator (partial #3691) #3716's; no rung here).
• Does not calibrate anything: every new number is unmeasured and says so; lane 5's bars stay measured-on-Aurora and unmeasured-for-stock.
• Does not lift the stock 1.30 ceiling (v3 on PostgreSQL-target analysis engine: v2/v3 slices and the v1 lanes' residue (from #3542) #3691), add query attribution to the profile, or give the Aurora profile detector the peak-and-mean gate (lane 9's body is untouched; it is the next shared-file batch's item if the coordinator wants parity — reported).
• Does not guess a duty cycle for pre-V133 rows or the extension arm. NULL is the whole interval, flagged.
• Does not touch Lite, deprecated/Dashboard, install/, or CHANGELOG.md.

Test plan

New — executed here, not just compiled, through a net10.0 xunit.v3 harness (AssemblyName Darling.Tests, bin copied inside the worktree so the walk-up censuses resolve) against a throwaway timescale/timescaledb:2.28.1-pg18 on Docker context desktop-linux migrated to V133 (torn down):

• PgTargetSampledWaitTests (7): the arm's shape (own metric, pg_wait_sampling only, sampled_ms with NULL as the interval, the reset rule, CPU excluded, the scaffold, no 30000, routed through the root); the floored set; the detector reads (peak AND mean, exact_collections, unknown_sampled_collections) and body (fence, sit-outs, both statistics in every gate, own bar never the Aurora one, no UtcNow / engine_kind, reachable after the wave-3 detector); the bar's lineage (literal, unmeasured, NOT an alias, // lane 24); the key through every switch (ratio family, the three ramps equal to the Aurora profile's, the escape at 15σ / 9×, a fired sampled standout + a load sibling past 1.49, 14σ capped at 1.49, five amplifiers); the fold onto the dominant sampled wait and solo with no contributors, the reconciler routing on the predicate, the tool row; the advice (duty cycle, mean, contributors, no "engine measured" / "spent" / "aurora", the caveat and first-occurrence renderings, the shared composer routing).
• PgTargetWaitTests (+2, 2 updated): the honest fraction executed on arranged rows — 1,440 samples × 1,000 ms over 48 × 30 s watched = 1.0, the same rows with sampled_ms unknown = 0.10, sampled_ms_known 1 / 0, wait_source_interval_ms, the witness fraction unchanged; the advice's duty-cycle sentence on the honest fact and the caveat on the blind one; both-sources: the exact facts carry the flag, both reads carry the cross-count, no 30000 in SQL or code, the collector stamps sampledCollections > 0 ? 1 : 0.
• PgTargetSampledWaitLiveTests (gated, [Collection("live-postgres")], LiveStoreCleanup): a stock-stamped server with 31 days of one-minute pg_database_stats and 8,929 five-minute sampler collections (sampled_ms = 30000, 1,000 ms period; Lock:relation 3–5 samples a cycle with a ripple for 30 days, then 45 for the last 250 minutes; IO:DataFileRead 6; CPU/Running 30) and a BOTH server carrying the same sampler rows plus 241 exact one-minute collections. Asserts the collector alone: Lock:relation 1.5 over 48 × 30,000 ms watched against 48 × 300,000 ms of wall, witness_fraction 0.15 (Value == witness × 10, the exact number), sampled_ms_known = 1; the BOTH server's facts are the exact ones at 0.20 with sampled_suppressed_by_exact = 1 and no is_sampled. The baseline alone: a trustworthy robust bucket, median in 300–367 ms/s, MAD > 0; the Aurora metric empty; the BOTH server's sampled bucket populated (the arm is not what sits out). The detector alone: ANOMALY_PG_SAMPLED_WAIT_PROFILE at peak = mean = 1,700 ms/s, modified_z ≥ 15, mean_modified_z ≥ 5, contrib_Lock:relation = 2,160,000 and contrib_IO:DataFileRead = 288,000 exactly, no CPU contributor, WaitProfileFamilies → (Lock:relation, Lock); nothing for the BOTH server. Then the REAL analyze_server: Lock:relation roots at 1.5 (the Lock rollup yields), its advice says "the sampler was watching … of wall clock … per second watched … 1.5 of one backend" and never "the engine measured"; the anomaly card shares the relation card's incident_id, its advice says "estimated from sampling … led by Lock:relation … The sampler watched … mean 1700 ms/sec", next_tools names get_pg_wait_sampling, severity 1.3; get_analysis_facts source=pg_waits shows 4 facts all with is_sampled = 1, sampled_ms_known = 1, lineage 0, interval > watched.

Edited censuses: PgTargetAnomalyTests (served metrics 8 → 9 with the table row; anomaly keys 9 → 10; the two new *Sql consts in the detector roster; PgSampledWaitProfileFallbackMsPerSec in the Pg* lineage list), BaselineSupplyTests (8 → 9, the derived list, the prose), PgTargetMeasuredLineageTests (the new bar in still-unmeasured), PgTargetBetweenWavesV2Tests (10 → 11).

Executed here: the harness's full sweep — PgTargetSampledWaitTests, PgTargetWaitTests, PgTargetWaitLiveTests (lane 5's, unchanged numbers: its rows carry no sampled_ms and read as before), PgTargetAnomalyTests incl. the lane-9 e2e, PgTargetSharedSwitchRoutingTests, PgTargetThresholdLineageTests, PgTargetMeasuredLineageTests, PgTargetFactCollectorTests, PgTargetMcpSurfaceTests, BaselineSupplyTests, LocalClockBucketKeyTests, PgTargetClockTests (lane 20's, over my arm), PgTargetBetweenWaves*, PgTargetBlocking/Io/Write/Replication/Bloat/Vacuum/Temp/Knobs/Queries/Sessions/Posture*Tests, PgTargetNamedHopsTests, StoreSqlClockDisciplineTests, DocCommentHygieneTests, RepoFileAdoptionTests, CommentFilterAdoptionTests, LivePostgresCollectionHygieneTests, FactSourceRegistryTests, AnalysisPass*Tests, FactCollector*Tests, all live classes with DARLING_TEST_PG on the rig — 728 / 729 green; the one red is LiveCleanupConversionRatchetTests naming 29 teardowns in DarlingCliCommandsTests.cs, a file #3762 landed on dev minutes before this rebase and this PR does not touch (dev red under me, reported to the coordinator).

Builds, -c Debug -p:EnableWindowsTargeting=true, zero warnings: PerformanceMonitor.Analysis, PerformanceMonitor.Darling.Analysis, PerformanceMonitor.Darling.Storage, PerformanceMonitor.Darling.Service, Darling.Tests, deprecated/Dashboard.Tests; Lite.Tests — dev's one pre-existing xUnit2000 in LogTailOverlapThresholdPinTests.cs:95, untouched. First executed in CI, stated plainly: the Windows build job is the first execution of the Lite/Viewer censuses (nothing here is SQL Server-side; no Lite file changes).

Which component(s) does this affect?

• Lite
• Darling
• Lite Tests
• Darling Tests
• SQL collection scripts
• Documentation
• Full Dashboard (deprecated)
• CLI Installer (deprecated)

How was this tested?

Above. Throwaway TimescaleDB 2.28.1 / PostgreSQL 18 container on Docker context desktop-linux, torn down. No production store or fleet server was read.

Checklist

• I have read the contributing guide
• My code builds with zero warnings (dotnet build -c Debug)
• I have tested my changes against at least one SQL Server version (no SQL Server surface is touched; every change is inside the pg_ vocabulary the shared switches route by prefix — PgTargetSharedSwitchRoutingTests.SqlServerAnomalies_AreUnaffected_ByThePgMembershipArm executed green)
• I have not introduced any hardcoded credentials or server names

Partial for #3691 (lane 24). CHANGELOG deliberately untouched — the coordinator splices this wave's entries.