Repository navigation
get_fleet_overview's collection-health rollup times out at :35 on the largest store — phase-grid signature, discriminate compression-band contention vs plan before fixing #3735
Description
get_fleet_overview times out at :35 on the largest store — three for three, retry always succeeds
Observed on the largest production store over the first day of the 3.8.0-nightly.20260919.450 build: the fleet-overview tool's collection-health rollup was cancelled by the server's statement_timeout at 06:35, 11:35 and 12:35Z — the store log carries the cancellation with the statement (v_collection_log, GROUP BY server_id, collector_name). Every retry succeeded. Three occurrences at the same minute is a phase-grid signature, not a slow-query one.
What is VERIFIED at source (dev at f5cc82e)
- The rollup is time-bounded:
DarlingFleetReader.cs:434–437—FROM v_collection_log WHERE collection_time >= $1 AND server_id <> 0 GROUP BY server_id, collector_name. The "unbounded history scan" reading is refuted; the window is the caller's. - The read runs under
McpCommandDeadlines.ReadSeconds = 20(McpCommandDeadlines.cs:130), so a 20-second budget is what the server cancelled it at. collection_logis a hypertable with its own compression policy that occupies a background-worker slot (TimescaleSupport.cs:39, 142–144).- The twenty continuous aggregates join the compression ladder: compress_after derived from each tier's refresh window, a once-a-day band off the hourly grid, and the backlog staged one aggregate per night, largest first (#3581) #3611's aggregate-compression band is pinned to :35Z, one aggregate per hour 01–20Z, and 2026-09-19 was its first live day (enabled at the 04:0x install; first run 01:35Z the following night per the install announce — but the hourly band ran through today).
HYPOTHESIS to discriminate before any fix
The :35 minute is where the once-a-day aggregate-compression band lands, and that band compresses the heaviest materializations first. A 20-second rollup over collection_log across 43 servers that normally finishes in a few seconds gets pushed past its budget by the compression job's I/O and lock-manager load, not by its own plan. Two cheap discriminators from the store side:
- Does the timeout recur only in hours when a compression job was running at :35 (
timescaledb_information.job_history— read as the owner role, per get_store_metrics: the job_history block said 'recording' and never said whose eyes the rows are visible to — it now evaluates the view's ownership filter for its own reader and proves rows exist where it can (#3574) #3585 — orpg_stat_activityshowingcompress_chunkat that minute)? A 13:35 / 14:35 recurrence with a job running, and a quiet 21:35–00:35 (band idle), settles it. EXPLAIN (ANALYZE, BUFFERS)of the rollup with the tool's actual$1on a quiet minute vs at :35 — if the plan is the same and only the time differs, it is contention, not the query.
If the hypothesis holds, the fix is scheduling, not SQL: either the rollup phase moves off :35 (the tool is on-demand, so the practical arm is the fleet cards' background refresh cadence), or the band moves off the minute the dashboard's own reads use — the same class as #3678 (three heaviest hypertables on consecutive minutes), and should be decided WITH it, after #3678's own re-measure tonight. If the hypothesis fails (timeouts at :35 with no job running), then the rollup deserves the #3647 treatment — measure, then a covering index on collection_log (collection_time, server_id, collector_name) if the plan shows a heap scan over the window's chunks.
Not a lane until discriminated. Requested of the monitor seat: the two reads above from the store, scrubbed, on the bus.
Activity
erikdarlingdata commentedon Sep 19, 2026
Solved at source: self-amplified fan-out of a window-independent rollup
Live catch at 13:36:03Z on the largest store (monitor seat, bus 20260919T133817Z): the collection-health rollup was running three times concurrently — identical statement, same start, one per
get_fleet_overviewcall the caller had raced in parallel with differenthours_backvalues — each with two parallel workers, nine backends oncollection_login pure I/O waits while the collectors' flush band was mid-COPY. The slowest copy crossed themcprole's server-sidestatement_timeout = 15 s; the retry landed after the flush drained and succeeded. Hit-vs-miss hours track write-band intensity, which explains 3 of 9.Cleared by the evidence: the plan (680 ms solo, healthy parallel aggregate); the #3611 compression band (wrong tables, ~0 s runs); the 15 s cap (doing its job — raising it would hide the fan-out).
What is VERIFIED at source
GetFleetOverview(hours_back)(DarlingMcpFleetTools.cs:64) →GetFleetOverviewAsync→ReadFailingCollectorCountsAsync(DarlingFleetReader.cs:1266–1272), which runsFleetCollectionHealthSqlover a fixed 7-day window (now.AddDays(-7)). Thehours_backargument bounds only blocking/deadlock; the rollup's result is identical for every window the caller asks for. Three calls with three windows = three identical 7-day scans.DarlingWebEndpoints.cs:263), so browser auto-refresh + an agent's tick can stack the same way.Fix shape (lane's choice within this)
Single-flight + short memo on the 7-day collection-health rollup inside
DarlingFleetReader: concurrent callers share one in-flight scan; a result younger than the collection cadence (60 s is the floor of what could have changed — the rollup counts per-collector runs at 1–5 min cadences) is served from memory. The other reads in the overview stay as they are; the memo is process-local, so the MCP host and the web host each hold their own.EXPLAINfrom the quiet arm is the acceptance: one scan per minute per host regardless of how many overview calls race.Caller-side hygiene, independent of the fix: an agent tick that wants three windows should call once with the widest and derive, or sequence the three — the rollup's cost is the same either way and the write band does not care about the caller's reasons.
get_fleet_overview: one collection-health scan per minute per host, h…
erikdarlingdata commentedon Sep 19, 2026
Closed by the merge of #3738 (its body names this issue with a closing keyword; keywords do not fire against dev, so the watcher closes it).
RepoFileAdoptionTests: FleetCollectionHealthMemoTests joins the LF-re…
CHANGELOG: second splice - sixty-three entries from sixty-five merged…
erikdarlingdata commentedon Sep 19, 2026
Live verification on the
.453fleet (17:1xZ): three consecutiveget_fleet_overviewcalls on one store readcollection_health_age_seconds= 0 → 4 → 7 — one fresh scan, then two memo-served reads ageing honestly. The single-flight holds.Reader note, not a defect: on the largest store the overview payload is ~1.3 MB and
collection_health_age_secondsserializes LAST, so an MCP wrapper with an output guard truncates before reaching it. The field was placed trailing for wire compatibility; the next lane that touchesFleetOverviewResult's serialization order should move the stamp near the head (additive fields are order-free for JSON consumers, so this costs nothing on the wire) — noted here so it isn't rediscovered.The per-server collection-health read is the fleet rollup's unmemoize…
A collector that produced and then stopped no longer reads HEALTHY: S…
A collector that produced and then stopped no longer reads HEALTHY: S…