The alert pass retries a store read once, two seconds later, before counting it failed — a write band's twelve-second stall no longer blinds an alert (#3848)

[erikdarlingdata]

The deadline is right; having no second attempt was the defect

DarlingAlertReadAdapter.AlertPassCommandTimeoutSeconds = 10 is the one value in its family deliberately set below what it inherited, and #2882's reasoning still holds in full: this pass has no enclosing CancelAfter, so the per-command deadline is the pass budget multiplied by however many reads run in sequence, and a read that runs long holds one of only MaxConcurrentServerSweeps fleet permits while the sweep skips relaunch for that server. Erring short is correct. Nothing here moves that number.

What the week's evidence adds is the shape of the failures it produces. Every alert-read kill on the largest production store was a sparse transient inside one of the store's own write bands — the fleet-overview fan-out racing into the flush band, the raw-hypertable compression band at steady state, timed-checkpoint fsync tails of 14–25 s, the daily-tier materialization's WAL storm, and the third band: the hourly successor refreshes materializing ~2 M rows/hour at evening peak with checkpoint syncs running 8–25 s chronically across the hour. Three client-deadline cancels in the 19Z hour alone, and the rate scales with the evening workload.

Each one is measured in seconds. The monitoring seat's own reads at the same minutes prove the point: its retry-once-sequential discipline lost no read all week. The alert pass had no retry at all, so a read that crossed 10 s was recorded by AlertReadFailureCounter (#3013), its condition was skipped for that cycle, and the next pass 30 s later ran into the same band still on.

So: re-ask once, after two seconds, on a command timeout and nothing else.

One seam, not fourteen edits

This type issues nineteen commands across twelve public reads, and the retry decision is one decision — fourteen copies of it would be free to disagree the moment any of them grew an arm, which is the argument PostgresTransportFault already makes for the write side's two callers. ExecuteWithOneRetryAsync takes each read whole as a delegate, so the retry re-runs from the read's connection-open outward rather than re-executing a half-consumed command. Every public read became a thin forwarder; each body moved verbatim to a private …CoreAsync sibling, SQL and parameters byte-identical.

Re-running a whole read is sound here because this type performs no accumulating write at all: the multi-command reads (database state's four maintenance statements ahead of its deviation read, anomalous jobs' freshness probe ahead of its page) are INSERT … ON CONFLICT DO NOTHING, an UPDATE to a fixed value, a DELETE of rows absent from the newest snapshot, and reads. That is a property of this file rather than a general licence — the same test StoreWriteReattempt.IsSafeToReattempt applies one seam over and answers differently, which is why its StoreCopyPhase conjunct is not reused: it is about a COPY's exactly-once property and has no meaning for a read.

The discrimination is the risky part, and it was measured rather than assumed

The gate must catch a client-side deadline and refuse three neighbours. The exception text cannot do this — Npgsql renders a client deadline as "Exception while reading from stream" with no SQLSTATE, identical to a dropped connection, which is the misdiagnosis #2826 exists to prevent. So the shapes were probed against the shipped Npgsql 10.0.3 on a throwaway postgres:17-alpine rig (SELECT pg_sleep(30) at CommandTimeout = 2):

case	measured chain	retried
client CommandTimeout expiry	NpgsqlException("Exception while reading from stream") → TimeoutException("Timeout during reading attempt"), no SQLSTATE	yes
…from ExecuteScalarAsync, ExecuteReaderAsync, and ReadAsync mid-stream	identical chain in all three	yes
server statement_timeout = 1500 under CommandTimeout = 60	bare PostgresException, SQLSTATE 57014, no TimeoutException anywhere	no
caller token cancelled mid-statement	OperationCanceledException → PostgresException 57014 ("user request")	no
pre-cancelled token	bare OperationCanceledException	no
connect timeout	NpgsqlException("Failed to connect to …") → TimeoutException("Timeout during connection attempt")	yes, deliberately

IsCommandTimeout walks the chain with the PostgresException test first at every level, so a server reply wrapped in transport machinery answers false. It deliberately does not end in "any NpgsqlException" the way the transport predicate does — a bare one with no TimeoutException inside it is some other client fault, and a gate ending in a catch-all would drift into retrying everything the first time Npgsql adds a wrapper. It is also narrower than IsTransportFault on purpose: a reset socket and a torn stream are connection faults rather than a read taking too long, and a broken connection during an alert pass is a condition the count should keep saying out loud. The connect timeout is on the retry side because its chain is indistinguishable from a command deadline except by message text, and a store unreachable inside its connect timeout is the same transient population worth one more ask.

Two further measurements make the second attempt sound rather than a read against a poisoned socket: after every deadline case the connection is left State = Open and immediately reusable — including a second ExecuteReaderAsync on the same connection after a mid-stream kill — and the backend's statement is genuinely cancelled (pg_stat_activity held zero surviving pg_sleep backends one second later).

Counting both truths

AlertReadFailureCounter gains RecordRetriedRead and a RetriedReads count beside ReadFailures. A retried-and-succeeded read increments only the retry count — the condition was judged, on evidence that arrived late. A read that stalls twice increments the retry count and, through the caller's existing catch arm exactly as before, today's failure count.

That second case is a decision the pin records and documents: retries are counted per attempt-pair, not per success. A retry counted only when it worked would make the figure a success rate and would understate the write bands' cost by precisely the episodes where the band was worst — the confident-zero shape #3013 exists to remove. And the seam records no failure itself, so a stalled-twice read is one failure, never two, with the caller's elapsed still measuring its own last attempt.

alert_read_health gains retried_reads and instance_retried_reads as trailing fields on both SKUs' get_collection_health and as two tiles on the web panel. They carry no stamp, read name or elapsed, unlike every count above them — those three date and attribute a condition that went blind, and a retried read did not, so a trio would invite reading a retry as a soft failure. On Lite both read zero as a property of the SKU rather than of a quiet store: Lite's alerting reads carry no command deadline, so there is no deadline to cross and nothing to retry, and the note says exactly that.

What this does NOT do

• It does not move the deadline. The asymmetry still says err short.
• It does not touch the grid redesign. The compression band puts the three heaviest hypertables on consecutive minutes: a 3.5-minute triple overlap starves alert reads nightly — the grid spaces by list position, not by measured runtime #3678 / Aggregate compression starts at :35 on every store every hour — the same minute the fleet's readers read; a 7-day scan races a compression start by design #3781 / Daily band re-derivation is DEFERRED until a daily successor is added (band full 23/23) — the slicing arm landed in #3865 (buckets_per_batch = 1); the storm class was fixed by the WAL ceiling (#3802) #3745 decide when the store writes; this decides whether a twelve-second write burst blinds an alert. No coupling either way — a rescheduled band makes the retry rarer, not wrong.
• It does not widen past the store's own reads. ServiceCommandDeadlines, TimescaleSupport and the alert engine's SQL Server-target reads are untouched; those are target reads, not store reads.
• It leaves the evaluator's own commands unretried, named below rather than half-done.

The new arithmetic is stated in the constant's doc: a stalled read costs at most 22 s instead of 10 s, and the pass's unbudgeted worst case becomes 45 × 22 s only when every read on every server stalls twice — a fleet-wide store outage, which is exactly the case where this pass should be slow. TheRetryPause_StaysInsideThePassesOwnArithmetic pins that two attempts plus the pause still fit inside the 30 s sweep interval, so the "one cycle of delay" cost the deadline's asymmetry is argued from stays true.

Which reads were routed, and which were not

Routed — all twelve public reads on DarlingAlertReadAdapter: blocking (XE + DMV fallback), blocking wait time, deadlocks, poison waits, long-running queries, database file growth, volume free space, PVS pressure, TempDB space, anomalous jobs, database state, forced-plan failures. The name each is given at the seam is the same one the engine's own catch arm records a failure under, asserted from both sources so one read cannot carry two names across the two counts.

Not routed, deliberately — the seven commands DarlingSelfAlertEvaluator executes on the same constant (ReadCollectionSignalsAsync, ReadMissingCaptureSessionsAsync, HasAgentEverBeenSeenRunningAsync, ReadLatestAgentStatusAsync, ReadLatestAgReplicaStatesAsync, ReadLatestAgDatabaseReplicaStatesAsync). These are internal static methods taking an NpgsqlDataSource rather than instance methods on the adapter, with their own live-fixture pins calling them statically — routing them needs either a static seam duplicating the instance one, or threading the counter and delay through six signatures and their tests. That is more than a mechanical route, so they stay unretried and are recorded here as the follow-up. The two heaviest of them were already fixed at the read (#3496, #3597 made both chunk-orderable), which is why they are the cheaper half to leave.

Tests

Darling.Tests/AlertReadRetrySeamTests.cs, fourteen cases, all executed locally:

• (a) a read that stalls once then answers → one result, RetriedReads = 1, ReadFailures = 0, and exactly one wait of exactly AlertPassRetryDelaySeconds — asserted as a value through an injected delay rather than by sleeping two real seconds.
• (b) stalls twice → the second attempt's exception propagates (not chained onto the first), RetriedReads = 1, and the seam records no failure because that is the caller's arm.
• (c) a theory over seven non-deadline faults → first attempt only, no wait, nothing counted: the store's statement_timeout, an ordinary server error, a server reply wrapped in a transport exception, a reset socket, a torn stream, a bare NpgsqlException, an unrelated fault.
• (d) the pass token cancelled strictly inside the wait → OperationCanceledException, no second attempt, and the retry still counted because it was already spent. Plus both measured first-attempt cancellation shapes, which the arm ahead of the filter refuses.
• (e) IsCommandTimeout against every measured chain in both directions, including a PostgresException outside a TimeoutException (a chain Npgsql does not produce today — pinned so the walk's correctness cannot come to depend on wrapping order) and the explicit disagreement with PostgresTransportFault.IsTransportFault on a reset socket.
• (f) EveryPublicReadOnTheAdapter_GoesOutThroughTheSeam, derived from source: every public read must be a non-async forwarder calling the seam with a private …CoreAsync sibling, so a thirteenth read written in the old shape reds on the day it is written rather than shipping unretried and silently blind. Its count floor earned its place immediately — the first cut of its regex stopped at the first > and saw four of twelve reads, which is the guard-sees-a-third-of-its-population failure this file exists to assert against.

Updated existing pins: both SKUs' AlertReadFailureSurfaceTests (record members 14 → 16, rendered field set 16 → 18 with the two new names, payload fields 14 → 16), and the trio rule now asserts the retry pair is outside it in both directions rather than relying on a suffix filter to miss it.

Full census run green post-rebase: AlertReadRetrySeamTests 14, AlertReadFailureSurfaceTests 27, AlertPassCommandTimeoutTests 11, RepoFileAdoptionTests 2, McpPageContractTests 43, McpPayloadContractCensusTests 67, MeasurementContractCensusTests 10, FactSourceRegistryTests 5, StragglerCommandTimeoutTests 48, ViewerCommandTimeoutTests 57 — 0 failures. Whole solution -c Release -p:EnableWindowsTargeting=true: 0 warnings, 0 errors.

Closes #3848