Skip to content

PostgreSQL blocking family: an operator now learns who headed a blocking chain, in what state, and how long the sessions behind it had really waited — PG_BLOCKING_CHAIN from the sampled edges, PG_LOCK_WAIT_EVENTS from the engine's own log lines, PG_LONG_RUNNING_QUERY from the active sessions, and a blocked-sessions anomaly whose baseline counts a quiet capture as zero (partial #3691, lane 17) - #3750

Merged
erikdarlingdata merged 1 commit into
devfrom
feat/3691-pg-blocking
Sep 19, 2026

Conversation

@erikdarlingdata

Copy link
Copy Markdown
Owner

What does this PR do?

Headline mechanism. A PostgreSQL operator now learns who was at the head of a blocking chain, in what state, and how long the sessions behind it had actually been waiting — PG_BLOCKING_CHAIN, reconstructed per capture from the sampled pg_blocking_edges (a HEAD is a blocker that is not itself blocked; the sessions behind it are every waiter reachable through the edges; depth is the longest path), named for the head with the most sessions behind it (application as role, database), graded on the blocked side's own blocked_query_duration_ms — never on captures × cadence — and routed by the head's STATE: an idle-in-transaction head walks to PG_IDLE_IN_TRANSACTION (the parked transaction lane 14 names), an active head to PG_LONG_RUNNING_QUERY when the pids match. Beside it, two facts the sample cannot make: PG_LOCK_WAIT_EVENTS from the engine's own log_lock_waits lines in pg_log_events (event grain — the count is complete and the duration is the engine's), and PG_LONG_RUNNING_QUERY from pg_session_states' active client rows (the active-query half of §2a, Lock waiters excluded because they are the chain's). And ANOMALY_PG_BLOCKING: blocked sessions per capture against a 30-day hour-of-week baseline in which a capture the collection log says ran and found no edge is a ZERO, judged on the #3653 PAIR gate (peak AND window mean). Lane 17 of #3691 fills the six stubs #3737 laid.

PARTIAL for #3691 — checklist line: "Active-query facts; blocking facts + BlockingChainReconstructor port", and the "contention anomalies" half of "Object growth / contention anomalies" (object growth stays blocked on a per-database size collector). Do not close.

Boundary extensions (read these first)

  • PerformanceMonitor.Analysis/PgTargetRelationshipGraph.Saturation.cs (+3 AddEdge, 11 → 14): Lock:relation → chain, Lock → chain, idle → chain. The STEP brief's rule — declare an edge from the file that owns the SOURCE node — puts them here, not in the blocking file; the wait/sessions pins moved with them (PgTargetWaitTests two → three destinations, PgTargetSessionsTests idle destinations 3 → 4 and the count, PgTargetBetweenWavesV2Tests' per-Lock-key count 2 → 3), each saying why.
  • Darling/PerformanceMonitor.Darling.Storage/DailySummaryHorizon.cs (BaselineServingRawCollectors + "pg_blocking", one doc sentence): the baseline arm reads pg_blocking_edges, and BaselineSupplyTests derives the floored set from the arms' text — the stub's own doc named this edit as the day-one consequence of filling the arm. The set is keyed by COLLECTOR name (what the purge resolves); pg_blocking / pg_blocking_edges is the first member whose two names differ, so the test now maps table → collector through the catalog (and its "pg_blocking is not floored" pin became pg_lock_stats).
  • PerformanceMonitor.Analysis/Baselines/AnomalyThresholds.cs: the // lane 17 block the brief assigned (PgBlockedSessionsFloor 3, PgBlockedSessionsFallback 10, both unmeasured with the calibrating read named); PgTargetAnomalyTests' constant roster gains the two.
  • Censuses moved deliberately: PgTargetAnomalyTests (metric 7 → 8 with its (name, table) row; the stub-loop drops PgBlockedSessions; collection_log admitted by name in the baseline AND detector FROM/JOIN censuses with the collector census's argument — see "the zero rule" below; detector SQL roster + fence census eight → nine; the AnomalyThresholds roster); PgTargetSharedSwitchRoutingTests (the four Null stub lines → NotNull + the anomaly's delegation equality, as the stub comment said); PgTargetBetweenWavesV2Tests (item 6 re-pinned from "inert" to "routed"; the floored set 9 → 10). PgTargetFactCollectorTests is untouched.

What the family concludes, and from which table

Five reads on one connection, one try, the shared three-outcome degrade: (1) pg_server_config for deadlock_timeout (the ONE engine-defined line here — stated on every fact as context, never a bar: it is the engine's logging threshold, 1 s by default) and log_lock_waits; (2) every edge in the window from pg_blocking_edges without either statement text (a doubles-only fact cannot carry it; get_pg_blocking can — the row's next_tools); (3) the sample's denominator — captures_total — by alias to DarlingPgBlockingReader.PgBlockingCaptureCountsSql, the read get_pg_blocking reports it from, so fact and tool cannot disagree; (4) the lock_wait family of pg_log_events plus the log collector's SUCCESS runs; (5) the active client rows of pg_session_states at or over the WARNING floor ($4, the scorer's constant — the SQL carries no second copy).

The reconstructor (PgTargetFactCollector.ReconstructBlockingCapture, internal, unit-tested) ports the SQL Server BlockingChainReconstructor's IDEA and none of its code — no spid/ecid/monitor_loop, no 1900 sentinel, no episode dedupe; a capture is one moment's edge list. BFS from each head with a visited set: a waiter held by two blockers is counted behind both (the sample says both hold it up), a cycle reachable from a head is walked once, and a blocked pid no head reaches is a cycle in flight (cycle_captures, counted, never attributed — the deadlock family's story). Across the window: the deepest chain, the peak distinct blocked count, the head with the most sessions behind it summed over its captures (NAMED), and the most captures ANY head headed (recurring_head_captures, the amplifier's witness — the chronic head need not be the most-blocked one, lane 14's recurrence rule). All-cycle windows: no fact, logged.

Verified at source, against the brief: the lock_wait parser (PgLockWaitEventParser) lifts no metrics — duration_ms and relation_name are NULL on every lock_wait row today. The duration is read off the message (after ([0-9.]+) ms, which the redactor keeps because bare numbers in prose are PostgreSQL's), the relation off the stored CONTEXT (in relation "orders", an identifier after a noun), each through coalesce so a column a parser fills tomorrow wins. Four line shapes are told apart: still waiting (one per wait — the COUNT), acquired … after N ms (the wait's true end-to-end length — what grades), detected deadlock (counted, the deadlock family's), avoided deadlock.

The lineage shape of every bar

Bar Value Lineage
BlockingWaitWarningMs / BlockingWaitCriticalMs 30 s / 5 min of the blocked side's own statement duration; ONE vocabulary for the chain and the written events unmeasured — the 2026-09-19 calibration took no distribution over pg_blocking_edges / pg_log_events (approximate row counts read 0, the unanalysed-table trap); its one lock number is a wait FRACTION (lane 5's bar). Calibrating reads named on the constants.
LongRunningQueryWarningMs / CriticalMs 10 min / 1 h (spelled 6 × Warning — a per-statement duration, not the A7 hour-total gate the census forbids; the file states why) unmeasured
BlockingPersistenceCaptures 3, BlockingCoFireBoost ×1.2 amplifier gates, not bars unmeasured
DeadlockTimeoutDefaultMs 1 000 context on the facts, never a bar engine-defined (the shipped default, stamped deadlock_timeout_from_snapshot = 0 when assumed)
PgBlockedSessionsFloor 3 / PgBlockedSessionsFallback 10 distinct blocked sessions in one capture unmeasured

Every fact of the family stamps threshold_lineage = 0. Every fact self-gates to 0 below its warning bar (the sessions family's rule), so a sibling predicate can ask presence and not a second bar.

The zero rule (pg_blocked_sessions) and its cost, stated

V71's own warning: pg_blocking_edges is EMPTY on a healthy instance, so over the table alone a quiet month has no baseline and a server that blocks once a week has a baseline of blocking. The only place "looked and found nothing" is written is collection_log (the read get_pg_blocking takes its denominator from). The arm and the detector's window read share one shape: the log's pg_blocking SUCCESS minutes FULL OUTER JOINed to the edges' COUNT(DISTINCT blocked_pid) per minute (date_bin('1 minute', …, TIMESTAMP '2000-01-01') — the runner stamps rows at run START and the log row at run END, so the two never match by instant; a boundary straddle costs one extra zero rather than a lost sample). The bin is the row's collection_time, naive UTC, so Q6's local-clock keying lands under it unchanged: clean(collection_time, v) + the one RobustTierScaffold, no EXTRACT of its own, no positional parameter past $3 (pinned). Cost: 30 days of one server's collection_log — on the order of a million rows on a full schedule — through idx_collection_log_time, filtered to one collector; a second or two on the largest store, once an hour per server (the provider's CacheTtl), analysis path only. This is a second admission of collection_log into the analysis reads (the collector census made the first, for the same argument); both baseline and detector censuses pin it by name with the reasoning.

Why the PAIR gate. First PostgreSQL detector to pass its window mean (the AnomalyGate #3653 remarks name the content lanes as the callers that would): a blocked-session series is spiky by nature, one lock handoff caught mid-flight is a 1 against a baseline of zeros, and the peak alone would fire on every such minute of a long window. Consequence stated: a 4-hour storm inside a 24-hour anchored pass may not fire the anomaly — PG_BLOCKING_CHAIN grades it on duration regardless.

The co-fire gating

  • Chain amplifiers (each ×1.2, on the sibling's BaseSeverity, never Severity): one head in ≥ 3 captures; PG_LOCK_WAIT_EVENTS fired (an independent source agrees); ANOMALY_PG_BLOCKING fired (unusual for THIS server). All three: 1.0 → 1.6 (executed).
  • Events: the chain fired (the sample names the root the line cannot). Long runner: same pid + query_id in ≥ 3 captures.
  • Graph (blocking category): chain → idle gated on the chain's OWN head state AND the idle fact's verdict — an active head beside an unrelated parked holder stays two findings (pinned); chain ↔ runner both ways on the exact pid seam; events → chain, anomaly → chain on the chain's verdict; from the saturation file, Lock* → chain and idle → chain. No chain → Lock edge, deliberately (the file says why: the wait and the chain are two readings of one contention, and the wait's edges re-enter the saturation mesh).
  • Traversal, executed: chain 1.2 vs idle 1.0 → [chain, idle]; idle 1.2 vs chain 0.5 → [idle, chain]; active head + parked holder → two stories.

The event fact's four shapes — and when the silent three are emitted

Events present → graded. No lines: log_lock_waits = off → unavailable + reason_log_lock_waits_off; no snapshot names it → reason_log_lock_waits_unknown; on but the log collector logged no SUCCESS run in the window → reason_log_collector_silent; on, collector ran, nothing → no_events = 1 (a measured zero). The silent shapes are emitted only beside a chain fact from the same window. First cut emitted them always, and three other families' e2e pins (PgTargetFactCollectorTests' "a day of history returns exactly the registry fact", Queries 4 → 5, Knobs 13 → 14) said what that means: a row on every quiet PostgreSQL target's get_analysis_facts saying nothing about that server. The "quiet server, no facts" contract holds; beside a chain the operator learns whether the engine's record corroborates, could not, or was not read.

Advice

Value-stated, no folklore: "billing-worker as billing headed a blocking chain while idle in transaction — 720 sessions queued behind it, the longest blocked for 6 min 40 s"; the sample caveat on every chain card ("caught in 240 of the 240 pg_blocking captures … read from the blocked session's own clock, never from captures × cadence"); the remedy switches on the head's STATE (idle → the application, idle_in_transaction_session_timeout ROLLS THE WORK BACK, pg_terminate_backend(pid); active → the statement, committed batches, lock ordering — "the one change that removes the wait instead of moving it"). Levers with counter-objectives: lock_timeout (the waiter fails; retry storms), deadlock_timeout (logging vs detection latency vs the check's cost), log_lock_waits = on (one line per wait past the timeout — cheap on any server not already blocking constantly). statement_timeout for the runner (ROLLS IT BACK). No DDL; an index is "a possibility to verify, not DDL to paste" (D8). The anomaly composes through the shared ComposeDeviation with the family's own static block (built per call — cross-partial static order), never the SQL Server "Anomalous spike" composer (pinned).

What it does NOT do

  • Does not carry the head's statement text on the fact (metadata is doubles; the two string seams carry application/role and database) — the brief asked for it; it is structurally impossible and get_pg_blocking is the row's first next_tools entry. Same for the lock-wait fingerprint (the relation rides as ObjectName).
  • No chain → Lock edge; no PG_WAIT_LOCK* edits beyond the source-owner file. No MaintenanceShapeShift (v3). No production read; every bar unmeasured and said so. No CREATE INDEX. No Lite twin (D1). No rung (StorageVersion untouched). Object growth stays blocked on a per-database size collector.
  • Does not touch FactScorer, FactAdvice, InferenceEngine, RelationshipGraph, the reconciler, DarlingAnalysisService, DarlingWorker, PgTargetFactCollector.cs (root), PgTargetFactKeys.cs, any root dispatcher, or PgTargetFactCollectorTests.

Test plan

New Darling/Darling.Tests/PgTargetBlockingTests.cs (33 ungated + 1 gated): the reconstructor (head, depth, sessions behind, a waiter held by two blockers, cycles with and without a head, empty); ParseDurationMs; the three ramps (29 999 → 0 / 30 000 → 0.5 / 165 000 → 0.75 / 300 000 → 1.0; 65 000 on the events; 599 999 → 0 / 600 000 → 0.5 / 2 400 000 → 0.8 / 3 600 000 → 1.0) with threshold_lineage = 0 at every band; unavailable / no_events score 0; the amplifiers through the real ScoreAll (1.0 / 1.2 / 1.6; an under-bar sibling amplifies nothing); the declared edge sets, the head-state gate, the pid seam both ways, the three "into a fired chain" edges, the traversal both ways round and the split; the advice by exact string for the exit shape and per branch (idle vs active head, four event shapes, runner with/without its chain, anomaly trusted vs first-occurrence); the SQL by text (floor as $4, no 600000, the exclusions, the message parse, the two log denominators, the alias to the reader's capture count, five command sites, read order, the silent-shapes gate); the arm's Q6 discipline and the shared zero rule.

The gated exit criterion: 31 days of one-minute pg_blocking log rows (set-based generate_series) with an edge every fifth minute and two every fiftieth (2 s each — the "0–2" routine); a 4-hour window with a 3-deep chain under an idle head in every capture (blocked 400 s / 200 s / 100 s), the parked head and a 40-minute report in pg_session_states every 5 min (plus a Lock waiter over the runner floor, to prove the exclusion), 12 lock_wait lines (six still waiting on orders, six acquired … after 65000.500 ms) with NULL duration_ms as the parser writes them. Collector: chain Value 400, head pid 9000 idle, 720 behind it over 240 captures, captures_total 240 from the log (not 241 — the runner logs at run END; the test says so), depth 3, peak 3; events 6 waits / 12 lines / max 65 000.5 / 1.5 per hour / orders; runner pid 9500 query_id 12345, one runner (the Lock waiter excluded). Baseline trustworthy, mean ∈ [0.1, 0.4], median 0; anomaly peak 3, mean ≈ 2.99, baseline_low_quality 0, both sigmas ≥ 2. Through the REAL analyze_server anchored at the window's end: PG_BLOCKING_CHAIN at 1.6 with the exact headline, leaf PG_IDLE_IN_TRANSACTION (no idle card of its own), the merged next_tools; PG_LOCK_WAIT_EVENTS at 0.68 with the chain named; PG_LONG_RUNNING_QUERY at 0.96; ANOMALY_PG_BLOCKING sharing the chain's incident_id, family prose; every tool get_pg_*; get_analysis_facts shows three facts, all threshold_lineage = 0.

Executed here, not only compiled — a net10.0 harness (AssemblyName=Darling.Tests, the 50-file PgTarget census set incl. BaselineSupplyTests, PgTargetAnomalyTests, PgTargetSharedSwitchRoutingTests, PgTargetMcpSurfaceTests, PgTargetThresholdLineageTests, PgTargetMeasuredLineageTests, DocCommentHygieneTests, both Live* censuses) against a throwaway timescale/timescaledb:2.28.1-pg18 on Docker context desktop-linux, migrated by the fixture, torn down after: 669 / 670 green, the one red DocCommentHygieneTests.TheDerivedProjectListCoversEveryProjectInTheTree on the harness csproj's own presence (the artifact #3665 / #3688 / #3715 / #3737 each recorded). That run includes every DARLING_TEST_PG-gated e2e in the set, this lane's 34 / 34 among them. Builds, zero warnings of ours: PerformanceMonitor.Analysis, PerformanceMonitor.Darling.Analysis, PerformanceMonitor.Darling.Service, Darling.Tests, Lite.Tests (the one xUnit2000 in LogTailOverlapThresholdPinTests.cs:95 is pre-existing on dev), deprecated/Dashboard.Tests (-p:EnableWindowsTargeting=true on the test projects). CI's Windows job is the first execution of the rest of the repo's censuses. No production store or fleet server was read.

Which component(s) does this affect?

  • Lite
  • Darling
  • Lite Tests
  • Darling Tests
  • SQL collection scripts
  • Documentation
  • Full Dashboard (deprecated)
  • CLI Installer (deprecated)

How was this tested?

Above. Throwaway container only; no production read.

Checklist

  • I have read the contributing guide
  • My code builds with zero warnings (dotnet build -c Debug)
  • I have tested my changes against at least one SQL Server version (PostgreSQL-target surface; the SQL Server pass is byte-identical by construction — no shared switch changed, and the routing pins that proved v1's byte-identity stayed green)
  • I have not introduced any hardcoded credentials or server names

Partial for #3691 (lane 17). CHANGELOG deliberately untouched — the coordinator splices this wave's entries.

…ructed from the sampled edges and graded on the blocked side's own duration, PG_LOCK_WAIT_EVENTS from the engine's log_lock_waits lines, PG_LONG_RUNNING_QUERY from the active session rows, ANOMALY_PG_BLOCKING over a zero-honest blocked-sessions baseline; the Lock waits and the idle holder walk into the chain
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant