Repository navigation
pg_index_bloat has failed every run since #2618 shipped: the 200-index budget admits 286GB on target-a, and the timeout it dies on cannot be classified #2997
Description
#2617 closed on rig evidence (160 indexes, 56ms) with its own real-cluster acceptance criteria left for "tomorrow's run": SUCCESS, 1,517 rows, ~200 measured, largest first. Nobody checked tomorrow's run. The store has now answered it eleven times.
Live state (pgmon store, read 2026-09-05)
Every pg_index_bloat run that has ever executed on the #2617 target (target-a) is status=ERROR, rows=0, message Exception while reading from stream, daily from the first-ever run at 2026-08-25T18:14:21Z (the run #2617 quotes) through 2026-09-04T19:00:27Z — eleven for eleven, including all nine runs after #2618's fix was hash-verified onto this box (#2617's close comment records the deploy at 2026-08-26T00:24Z; every subsequent daily run still fails identically). last_success: null.
The other 49 pgmon targets are all NO_PERMISSIONS (pgstatindex absent — pgstattuple never created), last_success: null. So the collector has produced zero rows fleet-wide, ever, and #2617's own closing worry stands: the #2561 exact-bloat claim remains unverified against any production cluster.
Cost on the failing target: the run is not free. Around the 2026-08-30 failure the entire sweep froze for 316 seconds (18:29:35 → 18:34:51), every 1-minute collector gapping, resuming the second the ERROR landed — and the daily run time drifts ~5 minutes later each day, which is a ~300-second slot being consumed daily. That is the collector blocking the sweep body for its full timeout before dying.
Mechanism, two layers — both verified in source on dev
1. The work budget bounds the wrong dimension for this target. PgIndexBloatCollector.cs:176-181 chose the 200-index count deliberately — "200 rather than a byte budget … A count is legible" — with MeasureCeilingBytes = 20GB (line 63) bounding any single index and largest-first ordering choosing which 200. On target-a that legibility trade fails: reading just the top 80 indexes from the store, 3 exceed the ceiling (correctly excluded, 71GB) and 77 sub-ceiling indexes sum 286GB, topping out at 19.68GB each — an independent read by another lane got the same shape (57 indexes / 265GB under a slightly different cut). The 200 largest sub-ceiling indexes are therefore hundreds of GB of pgstatindex page reads in one LEFT JOIN LATERAL statement. The 300-second CommandTimeoutSecondsOverride (line 190) cannot cover that; the count budget only bounds work when count correlates with bytes, and this target is the counterexample. #2617's own diagnosis — "a per-index ceiling and a total-work ceiling are different things and I only built the first" — happened a second time, one dimension up.
2. The timeout the fix added can never be classified. #2618's remedy 3 wanted "a classified timeout rather than a dropped connection". But the PostgreSQL classification arm in DarlingWorker.cs is:
catch (PostgresException ex) when (
PostgresFaultOutcome(ex, collectorName, runtime.ConnectedDatabase) is { Status: not "ERROR" } outcome)A client-side command timeout surfaces as NpgsqlException wrapping TimeoutException — not PostgresException, no SQLSTATE — so it can never enter PostgresFaultOutcome and falls to the general catch, which writes "ERROR", 0, 0, 0 with null phase blocks and the raw seven-word transport message. Two side effects worth naming: the hardcoded zeros make a 300-second death look like an instant one (it misled two independent investigations toward "connection dies at open"), and the message carries no collector/database/duration context at all — compare the authored text every classified skip on this fleet gets.
Fix shape, in this order (order matters)
- Widen the PG fault classification to cover non-
PostgresExceptionfaults — an authoredCommandTimeoutoutcome ("client-side deadline of Ns expired; the statement was cancelled mid-read") and aConnectionFatalone — without disturbing the existing SQLSTATE path that correctly degrades the other 49 targets to PERMISSIONS. Doing 2 first would leave the next timeout as unreadable as this one. - Bound bytes, not just count: cap the cycle at a cumulative sub-ceiling byte budget (keep largest-first; keep
skipped_reason = not measured this cycle (work budget)for the remainder). The :176 comment's legibility argument can survive as "N indexes or B bytes, whichever first". - Record real elapsed time on the ERROR arm instead of literal zeros — the current shape destroys exactly the evidence that distinguishes a timeout from a connection failure.
The observation that decides
Next scheduled attempt on this target is ~2026-09-05T19:05Z (daily cadence drifting from 18:14 → 19:00 across the eleven runs — note it is NOT in the ~10:33Z block the other dailies use), and it will be the first attempt on the build deployed 2026-09-05 04:13Z. Expected under this diagnosis: identical failure, since nothing in that build touches the budget or the classification. A SUCCESS there falsifies layer 1 and leaves only layer 2 to fix.
Related: #2617 (closed on the rig), #2618 (the fix this measures), #2561 (the claim still unverified in production), #2994 (the same rollup-honesty family on this fleet), #2623.
Activity
[-]pg_index_bloat has failed every run since #2618 shipped: the 200-index budget admits 286GB on looker-poc, and the timeout it dies on cannot be classified[/-][+]pg_index_bloat has failed every run since #2618 shipped: the 200-index budget admits 286GB on target-a, and the timeout it dies on cannot be classified[/+]Move the two DarlingWorker source pins the new timeout arm shifts
Stop the timeout message naming a knob most collectors do not have
Say in the query that there are two budgets now
Say why the timeout filter passes yieldsOnLockTimeout: false
Pin that the cycle budget never drops below the per-index ceiling
Say what the explicit ROWS frame actually buys, measured
Name the database that actually faulted, not the one the probe landed on
Point at the constant, not at a line number
Let only an Npgsql fault earn the Npgsql narrative
Bound pg_index_bloat by bytes and classify the deadline it dies on (#…
erikdarlingdata commentedon Sep 5, 2026
Closed by #3003, merged to
devas7a25aab367.What landed — five changes, not the three this issue scoped
LEFT JOIN LATERAL pgstatindex(...)join condition itself rather than on anyskipped_reasonpath. It charges only sub-ceiling indexes, matching the LATERAL's own ceiling term — load-bearing, because the three over-ceiling indexes total 71 GB against a 20 GB budget and charging them would spend the whole budget on work never done. No starvation band: the gate is sum-including-current-row<=budget with budget==ceiling, so the first sub-ceiling index always fits.PostgresFaultOutcome's parameter type, so no amount of SQLSTATE widening could ever have reached it. Status staysERRORdeliberately: a deadline that collected nothing belongs in error counts and the health band.ConnectionFatalstays with the general catch, which owns the reprobe.ServerRuntime.ConnectedDatabase, stamped once at probe, so every fault reported the wrong database. Both now route throughCollectorFaultDatabase.For(ex, fallback). This also fixed A missing-extension message says CREATE EXTENSION without naming which database, and the fleet has one installed in the wrong one #2638's extension message, which carried the same defect on all 49 degraded targets.TimeoutExceptionwhile the authored message claimed an Npgsql mechanism.Items 4 and 5 came from review, not from this issue.
An amplification this issue understated
RunsPerDatabaseis true, the per-database loop continues past a failure (failed++ / continue), and the rethrow fires only atfailed == attempted. So the deadline is paid per database, not once per sweep. 316 s therefore implies one attempted database — it is not the ceiling. The widest clusters on this fleet reach 16 attempted databases, which is roughly 80 minutes of worst-case sweep freeze before this fix.The evidence for the original diagnosis was destroyed at write time
The ERROR arm called
LogCollectionAsync(..., "ERROR", 0, 0, 0, ...)andduration_msis written assqlMs + storageMs, so all eleven failing rows carryduration_ms = 0. The 316 s sweep freeze — inferred from other collectors' gaps — is the only surviving measurement. The 300 s timeout is established from source, not from a stored duration.The six classified non-ERROR arms still write literal zeros and were deliberately left alone, because
collector_costand the health bands currently treat classified skips as free, and making them carry real duration changes what those surfaces report for degraded servers. That is a choice someone should make rather than inherit, and it is now its own work with the read-surface decision required before any write change.Verification
Nine mutations, one at a time, anchor count and source hash each. A comment-only control moved both the source hash and the assembly hash while failing nothing, which is why neither is cited as evidence of behaviour. The shipped
BuildQuery(ctx).Textwas run verbatim against PostgreSQL 17.11 and 16.15 —Darling PostgreSQL testsexercises the store, and CI never parses collector SQL, sosum(...) FILTER (...) OVER (...)had no other route. TheFILTER's negative control produced the predicted wrong answer, and theROWS/RANGEtie was measured after reasoning got the direction backwards.Not verified
Nothing ran against the affected target. The deciding observation is its first daily run after the next nightly reaches the box — a failure before then is the currently-deployed build, not this fix. Sweep-freeze detachment is deliberately out of scope: an independent change, and if the budget works there is nothing to detach around.
erikdarlingdata commentedon Sep 5, 2026
Tonight's daily run fired at 19:05:33Z and failed — but this is a pre-fix control, not the deciding observation, and it should not be read as the budget gate failing.
Build provenance, checked before interpreting the run: the monitoring host's service binary is 3.6.0 with a write time of
2026-09-05T03:59:00Z. #3003 (the byte budget plus the classified deadline) merged at2026-09-05T13:48:22Z, roughly ten hours after that binary was produced, and the most recent published nightly asset is dated06:27:28Z— also before the merge. So the running build cannot contain the fix, whichever way the nightly was installed.The run itself, from the store's collection log on the monitoring host:
Exception while reading from streamOne row for the whole cycle rather than the per-database fan-out the collector normally writes, and the service log attributes it to the Aurora target this issue is about. That is the same shape the issue was filed on: the read dies against the target and the cycle stores nothing.
Worth separating from a second, unrelated condition in the same collector's history. The previous day's runs against other targets are
PERMISSIONSwithfunction public.pgstatindex(regclass) does not exist (SQLSTATE 42883)— that ispgstattuplenot being installed on those clusters, not a deadline problem, and no budget change affects it. Only theERROR/stream-exception rows belong to this issue.So the deciding observation is still ahead, and it needs the first daily run after a build containing #3003 actually reaches the host. Two ways to read that run when it comes:
The second reading is only available on the new build, which is another reason not to score tonight's failure against the fix.
Rotate which indexes pg_index_bloat measures, so its deferral claim i…
Take #3164's and #3166's entries from the lane handoff files instead …
Record the CHANGELOG entries for thirty-three changes across twenty-s…
Tell a permissions fault from a missing log file, and logging_collect…