Repository navigation
PostgreSQL blocking family: an operator now learns who headed a blocking chain, in what state, and how long the sessions behind it had really waited — PG_BLOCKING_CHAIN from the sampled edges, PG_LOCK_WAIT_EVENTS from the engine's own log lines, PG_LONG_RUNNING_QUERY from the active sessions, and a blocked-sessions anomaly whose baseline counts a quiet capture as zero (partial #3691, lane 17) - #3750
Merged
Merged
Conversation
…ructed from the sampled edges and graded on the blocked side's own duration, PG_LOCK_WAIT_EVENTS from the engine's log_lock_waits lines, PG_LONG_RUNNING_QUERY from the active session rows, ANOMALY_PG_BLOCKING over a zero-honest blocked-sessions baseline; the Lock waits and the idle holder walk into the chain
erikdarlingdata
enabled auto-merge (squash)
September 19, 2026 15:29
54 of 55 tasks
This was referenced Sep 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Headline mechanism. A PostgreSQL operator now learns who was at the head of a blocking chain, in what state, and how long the sessions behind it had actually been waiting —
PG_BLOCKING_CHAIN, reconstructed per capture from the sampledpg_blocking_edges(a HEAD is a blocker that is not itself blocked; the sessions behind it are every waiter reachable through the edges; depth is the longest path), named for the head with the most sessions behind it (application as role, database), graded on the blocked side's ownblocked_query_duration_ms— never on captures × cadence — and routed by the head's STATE: an idle-in-transaction head walks toPG_IDLE_IN_TRANSACTION(the parked transaction lane 14 names), an active head toPG_LONG_RUNNING_QUERYwhen the pids match. Beside it, two facts the sample cannot make:PG_LOCK_WAIT_EVENTSfrom the engine's ownlog_lock_waitslines inpg_log_events(event grain — the count is complete and the duration is the engine's), andPG_LONG_RUNNING_QUERYfrompg_session_states' active client rows (the active-query half of §2a, Lock waiters excluded because they are the chain's). AndANOMALY_PG_BLOCKING: blocked sessions per capture against a 30-day hour-of-week baseline in which a capture the collection log says ran and found no edge is a ZERO, judged on the #3653 PAIR gate (peak AND window mean). Lane 17 of #3691 fills the six stubs #3737 laid.PARTIAL for #3691 — checklist line: "Active-query facts; blocking facts +
BlockingChainReconstructorport", and the "contention anomalies" half of "Object growth / contention anomalies" (object growth stays blocked on a per-database size collector). Do not close.Boundary extensions (read these first)
PerformanceMonitor.Analysis/PgTargetRelationshipGraph.Saturation.cs(+3AddEdge, 11 → 14):Lock:relation → chain,Lock → chain,idle → chain. The STEP brief's rule — declare an edge from the file that owns the SOURCE node — puts them here, not in the blocking file; the wait/sessions pins moved with them (PgTargetWaitTeststwo → three destinations,PgTargetSessionsTestsidle destinations 3 → 4 and the count,PgTargetBetweenWavesV2Tests' per-Lock-key count 2 → 3), each saying why.Darling/PerformanceMonitor.Darling.Storage/DailySummaryHorizon.cs(BaselineServingRawCollectors+"pg_blocking", one doc sentence): the baseline arm readspg_blocking_edges, andBaselineSupplyTestsderives the floored set from the arms' text — the stub's own doc named this edit as the day-one consequence of filling the arm. The set is keyed by COLLECTOR name (what the purge resolves);pg_blocking/pg_blocking_edgesis the first member whose two names differ, so the test now maps table → collector through the catalog (and its "pg_blocking is not floored" pin becamepg_lock_stats).PerformanceMonitor.Analysis/Baselines/AnomalyThresholds.cs: the// lane 17block the brief assigned (PgBlockedSessionsFloor 3,PgBlockedSessionsFallback 10, both unmeasured with the calibrating read named);PgTargetAnomalyTests' constant roster gains the two.PgTargetAnomalyTests(metric 7 → 8 with its (name, table) row; the stub-loop dropsPgBlockedSessions;collection_logadmitted by name in the baseline AND detector FROM/JOIN censuses with the collector census's argument — see "the zero rule" below; detector SQL roster + fence census eight → nine; the AnomalyThresholds roster);PgTargetSharedSwitchRoutingTests(the fourNullstub lines →NotNull+ the anomaly's delegation equality, as the stub comment said);PgTargetBetweenWavesV2Tests(item 6 re-pinned from "inert" to "routed"; the floored set 9 → 10).PgTargetFactCollectorTestsis untouched.What the family concludes, and from which table
Five reads on one connection, one
try, the shared three-outcome degrade: (1)pg_server_configfordeadlock_timeout(the ONE engine-defined line here — stated on every fact as context, never a bar: it is the engine's logging threshold, 1 s by default) andlog_lock_waits; (2) every edge in the window frompg_blocking_edgeswithout either statement text (a doubles-only fact cannot carry it;get_pg_blockingcan — the row'snext_tools); (3) the sample's denominator —captures_total— by alias toDarlingPgBlockingReader.PgBlockingCaptureCountsSql, the readget_pg_blockingreports it from, so fact and tool cannot disagree; (4) thelock_waitfamily ofpg_log_eventsplus the log collector's SUCCESS runs; (5) the active client rows ofpg_session_statesat or over the WARNING floor ($4, the scorer's constant — the SQL carries no second copy).The reconstructor (
PgTargetFactCollector.ReconstructBlockingCapture, internal, unit-tested) ports the SQL ServerBlockingChainReconstructor's IDEA and none of its code — no spid/ecid/monitor_loop, no 1900 sentinel, no episode dedupe; a capture is one moment's edge list. BFS from each head with a visited set: a waiter held by two blockers is counted behind both (the sample says both hold it up), a cycle reachable from a head is walked once, and a blocked pid no head reaches is a cycle in flight (cycle_captures, counted, never attributed — the deadlock family's story). Across the window: the deepest chain, the peak distinct blocked count, the head with the most sessions behind it summed over its captures (NAMED), and the most captures ANY head headed (recurring_head_captures, the amplifier's witness — the chronic head need not be the most-blocked one, lane 14's recurrence rule). All-cycle windows: no fact, logged.Verified at source, against the brief: the
lock_waitparser (PgLockWaitEventParser) lifts no metrics —duration_msandrelation_nameare NULL on every lock_wait row today. The duration is read off the message (after ([0-9.]+) ms, which the redactor keeps because bare numbers in prose are PostgreSQL's), the relation off the stored CONTEXT (in relation "orders", an identifier after a noun), each throughcoalesceso a column a parser fills tomorrow wins. Four line shapes are told apart:still waiting(one per wait — the COUNT),acquired … after N ms(the wait's true end-to-end length — what grades),detected deadlock(counted, the deadlock family's),avoided deadlock.The lineage shape of every bar
BlockingWaitWarningMs/BlockingWaitCriticalMspg_blocking_edges/pg_log_events(approximate row counts read 0, the unanalysed-table trap); its one lock number is a wait FRACTION (lane 5's bar). Calibrating reads named on the constants.LongRunningQueryWarningMs/CriticalMs6 × Warning— a per-statement duration, not the A7 hour-total gate the census forbids; the file states why)BlockingPersistenceCaptures3,BlockingCoFireBoost×1.2DeadlockTimeoutDefaultMs1 000deadlock_timeout_from_snapshot = 0when assumed)PgBlockedSessionsFloor3 /PgBlockedSessionsFallback10Every fact of the family stamps
threshold_lineage = 0. Every fact self-gates to 0 below its warning bar (the sessions family's rule), so a sibling predicate can ask presence and not a second bar.The zero rule (
pg_blocked_sessions) and its cost, statedV71's own warning:
pg_blocking_edgesis EMPTY on a healthy instance, so over the table alone a quiet month has no baseline and a server that blocks once a week has a baseline of blocking. The only place "looked and found nothing" is written iscollection_log(the readget_pg_blockingtakes its denominator from). The arm and the detector's window read share one shape: the log'spg_blockingSUCCESS minutes FULL OUTER JOINed to the edges'COUNT(DISTINCT blocked_pid)per minute (date_bin('1 minute', …, TIMESTAMP '2000-01-01')— the runner stamps rows at run START and the log row at run END, so the two never match by instant; a boundary straddle costs one extra zero rather than a lost sample). The bin is the row'scollection_time, naive UTC, so Q6's local-clock keying lands under it unchanged:clean(collection_time, v)+ the oneRobustTierScaffold, no EXTRACT of its own, no positional parameter past$3(pinned). Cost: 30 days of one server'scollection_log— on the order of a million rows on a full schedule — throughidx_collection_log_time, filtered to one collector; a second or two on the largest store, once an hour per server (the provider'sCacheTtl), analysis path only. This is a second admission ofcollection_loginto the analysis reads (the collector census made the first, for the same argument); both baseline and detector censuses pin it by name with the reasoning.Why the PAIR gate. First PostgreSQL detector to pass its window mean (the
AnomalyGate#3653 remarks name the content lanes as the callers that would): a blocked-session series is spiky by nature, one lock handoff caught mid-flight is a 1 against a baseline of zeros, and the peak alone would fire on every such minute of a long window. Consequence stated: a 4-hour storm inside a 24-hour anchored pass may not fire the anomaly —PG_BLOCKING_CHAINgrades it on duration regardless.The co-fire gating
BaseSeverity, neverSeverity): one head in ≥ 3 captures;PG_LOCK_WAIT_EVENTSfired (an independent source agrees);ANOMALY_PG_BLOCKINGfired (unusual for THIS server). All three: 1.0 → 1.6 (executed).blockingcategory):chain → idlegated on the chain's OWN head state AND the idle fact's verdict — an active head beside an unrelated parked holder stays two findings (pinned);chain ↔ runnerboth ways on the exact pid seam;events → chain,anomaly → chainon the chain's verdict; from the saturation file,Lock* → chainandidle → chain. Nochain → Lockedge, deliberately (the file says why: the wait and the chain are two readings of one contention, and the wait's edges re-enter the saturation mesh).[chain, idle]; idle 1.2 vs chain 0.5 →[idle, chain]; active head + parked holder → two stories.The event fact's four shapes — and when the silent three are emitted
Events present → graded. No lines:
log_lock_waits = off→unavailable+reason_log_lock_waits_off; no snapshot names it →reason_log_lock_waits_unknown; on but the log collector logged no SUCCESS run in the window →reason_log_collector_silent; on, collector ran, nothing →no_events = 1(a measured zero). The silent shapes are emitted only beside a chain fact from the same window. First cut emitted them always, and three other families' e2e pins (PgTargetFactCollectorTests' "a day of history returns exactly the registry fact", Queries 4 → 5, Knobs 13 → 14) said what that means: a row on every quiet PostgreSQL target'sget_analysis_factssaying nothing about that server. The "quiet server, no facts" contract holds; beside a chain the operator learns whether the engine's record corroborates, could not, or was not read.Advice
Value-stated, no folklore: "billing-worker as billing headed a blocking chain while idle in transaction — 720 sessions queued behind it, the longest blocked for 6 min 40 s"; the sample caveat on every chain card ("caught in 240 of the 240 pg_blocking captures … read from the blocked session's own clock, never from captures × cadence"); the remedy switches on the head's STATE (idle → the application,
idle_in_transaction_session_timeoutROLLS THE WORK BACK,pg_terminate_backend(pid); active → the statement, committed batches, lock ordering — "the one change that removes the wait instead of moving it"). Levers with counter-objectives:lock_timeout(the waiter fails; retry storms),deadlock_timeout(logging vs detection latency vs the check's cost),log_lock_waits = on(one line per wait past the timeout — cheap on any server not already blocking constantly).statement_timeoutfor the runner (ROLLS IT BACK). No DDL; an index is "a possibility to verify, not DDL to paste" (D8). The anomaly composes through the sharedComposeDeviationwith the family's own static block (built per call — cross-partial static order), never the SQL Server "Anomalous spike" composer (pinned).What it does NOT do
get_pg_blockingis the row's firstnext_toolsentry. Same for the lock-wait fingerprint (the relation rides asObjectName).chain → Lockedge; noPG_WAIT_LOCK*edits beyond the source-owner file. NoMaintenanceShapeShift(v3). No production read; every bar unmeasured and said so. NoCREATE INDEX. No Lite twin (D1). No rung (StorageVersionuntouched). Object growth stays blocked on a per-database size collector.FactScorer,FactAdvice,InferenceEngine,RelationshipGraph, the reconciler,DarlingAnalysisService,DarlingWorker,PgTargetFactCollector.cs(root),PgTargetFactKeys.cs, any root dispatcher, orPgTargetFactCollectorTests.Test plan
New
Darling/Darling.Tests/PgTargetBlockingTests.cs(33 ungated + 1 gated): the reconstructor (head, depth, sessions behind, a waiter held by two blockers, cycles with and without a head, empty);ParseDurationMs; the three ramps (29 999 → 0 / 30 000 → 0.5 / 165 000 → 0.75 / 300 000 → 1.0; 65 000 on the events; 599 999 → 0 / 600 000 → 0.5 / 2 400 000 → 0.8 / 3 600 000 → 1.0) withthreshold_lineage = 0at every band; unavailable / no_events score 0; the amplifiers through the realScoreAll(1.0 / 1.2 / 1.6; an under-bar sibling amplifies nothing); the declared edge sets, the head-state gate, the pid seam both ways, the three "into a fired chain" edges, the traversal both ways round and the split; the advice by exact string for the exit shape and per branch (idle vs active head, four event shapes, runner with/without its chain, anomaly trusted vs first-occurrence); the SQL by text (floor as$4, no600000, the exclusions, the message parse, the two log denominators, the alias to the reader's capture count, five command sites, read order, the silent-shapes gate); the arm's Q6 discipline and the shared zero rule.The gated exit criterion: 31 days of one-minute
pg_blockinglog rows (set-basedgenerate_series) with an edge every fifth minute and two every fiftieth (2 s each — the "0–2" routine); a 4-hour window with a 3-deep chain under an idle head in every capture (blocked 400 s / 200 s / 100 s), the parked head and a 40-minute report inpg_session_statesevery 5 min (plus a Lock waiter over the runner floor, to prove the exclusion), 12lock_waitlines (sixstill waitingonorders, sixacquired … after 65000.500 ms) with NULLduration_msas the parser writes them. Collector: chain Value 400, head pid 9000 idle, 720 behind it over 240 captures,captures_total240 from the log (not 241 — the runner logs at run END; the test says so), depth 3, peak 3; events 6 waits / 12 lines / max 65 000.5 / 1.5 per hour /orders; runner pid 9500 query_id 12345, one runner (the Lock waiter excluded). Baseline trustworthy, mean ∈ [0.1, 0.4], median 0; anomaly peak 3, mean ≈ 2.99,baseline_low_quality0, both sigmas ≥ 2. Through the REALanalyze_serveranchored at the window's end:PG_BLOCKING_CHAINat 1.6 with the exact headline, leafPG_IDLE_IN_TRANSACTION(no idle card of its own), the mergednext_tools;PG_LOCK_WAIT_EVENTSat 0.68 with the chain named;PG_LONG_RUNNING_QUERYat 0.96;ANOMALY_PG_BLOCKINGsharing the chain'sincident_id, family prose; every toolget_pg_*;get_analysis_factsshows three facts, allthreshold_lineage = 0.Executed here, not only compiled — a net10.0 harness (
AssemblyName=Darling.Tests, the 50-file PgTarget census set incl.BaselineSupplyTests,PgTargetAnomalyTests,PgTargetSharedSwitchRoutingTests,PgTargetMcpSurfaceTests,PgTargetThresholdLineageTests,PgTargetMeasuredLineageTests,DocCommentHygieneTests, both Live* censuses) against a throwawaytimescale/timescaledb:2.28.1-pg18on Docker contextdesktop-linux, migrated by the fixture, torn down after: 669 / 670 green, the one redDocCommentHygieneTests.TheDerivedProjectListCoversEveryProjectInTheTreeon the harness csproj's own presence (the artifact #3665 / #3688 / #3715 / #3737 each recorded). That run includes everyDARLING_TEST_PG-gated e2e in the set, this lane's 34 / 34 among them. Builds, zero warnings of ours:PerformanceMonitor.Analysis,PerformanceMonitor.Darling.Analysis,PerformanceMonitor.Darling.Service,Darling.Tests,Lite.Tests(the one xUnit2000 inLogTailOverlapThresholdPinTests.cs:95is pre-existing on dev),deprecated/Dashboard.Tests(-p:EnableWindowsTargeting=trueon the test projects). CI's Windows job is the first execution of the rest of the repo's censuses. No production store or fleet server was read.Which component(s) does this affect?
How was this tested?
Above. Throwaway container only; no production read.
Checklist
dotnet build -c Debug)Partial for #3691 (lane 17). CHANGELOG deliberately untouched — the coordinator splices this wave's entries.