The compression band puts the three heaviest hypertables on consecutive minutes: a 3.5-minute triple overlap starves alert reads nightly — the grid spaces by list position, not by measured runtime #3678

Activity

1. erikdarlingdata commented on Sep 20, 2026

   

   erikdarlingdata

   on Sep 20, 2026

   OwnerAuthor

   More actions

   Night 1 on the new build: the discriminator came back near-zero, so this is a deferral, not a lane

   The issue's own sequencing rule was "lane only on a non-zero result" from the first post-install night's band. The full-band read for tonight (00:36–00:59Z, the first band carrying #3647's chunk-ordered capture-down read and #3586's Index Only Scan for the forced-plan read) on the largest production store:

   • Alert-side read failures inside the band: 0. The original victim class — capture-down and forced-plan alert reads dying at their 10-second deadline — went from 39 kills in the hour (the night this issue was filed) to none. That is the whole of the harm this issue named, and it is gone on the reads' own fix, before anything was done to the band.
   • One boundary kill, not in the victim class. The MCP overview's 7-day collection-health scan — the heaviest read class on the store — started 00:35:53 and died 00:36:08, exactly at the band boundary. Night 0 and night 1 carry one-time lifts (the install's cold caches, the first chunk close under the new refresh grid), and the kill did not recur inside the band, which is consistent with a lift rather than the overlap.

   So the triple overlap at :36–:38 is real and still costs compression time, but tonight it cost no alert coverage — the lever this issue reached for is, on tonight's evidence, optional hygiene.

   Disposition

   The instrument stays armed for night 2 (tomorrow, 00:36–00:59Z): the first band with no night-0 lifts in it.

   • If night 2 shows zero alert-read kills AND at most one boundary kill, this issue closes as resolved by The interval-hourly refresh recomputes one fleet-wide bucket per dirty hour and paid twelve index inserts per row for eleven indexes nothing reads; the capture-down alert read decompressed a server's whole collection_log to learn two statuses (#3597, partial) #3647 / The forced-plan-failures alert read gets the covering index the planner will actually take — the composite it needed was already there and priced out, so it walked the whole fleet's two-hour slice per server (#3573) #3586, with the boundary kill noted as covered by the get_fleet_overview: one collection-health scan per minute per host, however many overview calls race (#3735) #3738 memo (a retry of that scan within 60 s is served from memory, so a single boundary death costs one refresh, not a page).
   • If any alert read dies inside the band on night 2, the lane opens — with the smallest arm first: nudge the band's first minute from :36 to :37 (one constant — CompressionPhaseGuardMinutes, or the band's first-minute derivation in TimescaleSupport.BuildCompressionPhaseMinutes) so the aggregate band's tail and the raw band's head stop touching, rather than re-spacing three hypertables by measured runtime. The runtime-spacing design above stays on the record as the second arm if the nudge is not enough.

   #3745 (the daily band re-derivation) keeps its stated dependency on this issue's outcome either way.
2. erikdarlingdata commented on Sep 22, 2026

   

   erikdarlingdata

   on Sep 22, 2026

   OwnerAuthor

   More actions

   The three-band design — one decision, three issues (#3678, #3781, #3745), written here as the coupling home

   The monitoring seat's week of reads (bus 20260921T170059Z, 19:43Z, 21:40Z) settled what each band does to the largest store's readers. Every read-kill this week belongs to one of three write bands, and the fix for each is different in kind, so the decision is to name them and stop treating "the band" as one thing.

   band	what writes	when	measured effect	verdict
   A — nightly raw compression (#3678)	the 73 raw hypertables' daily policy_compression, assigned CompressionPhaseMinutes[index % 25] in REGISTRY order over the :35–:59 tail of hour 00	once a night	night 2 (steady state): query_stats 552 s at 00:40, query_snapshots 324 s at 00:41, query_store_stats beside them — eleven alert-read cancels 00:43–00:46; night 1's clean was a partial-day artefact	GUILTY — lane
   B — aggregate compression slots (#3781)	one continuous aggregate's daily compression per hour, minute :35 (= last minute of the heaviest refresh window, the minute the hourly grid holds and leaves unused)	01–23Z, one per hour	steady-state runs 68 s-class; the killer runs were the STAGED first lifts (1,246 s for query_store_stats_hourly at 03:35) — 23 of them, one per night, ending ~Oct 12; the exposed readers are anything on a 7-day scan at :35 sharp	count-only, with a re-open bar
   C — hourly successor refreshes (the "third band")	#3731's *_interval_hourly successors materializing ~340–700 K rows each per hour (~2 M rows/hour at evening peak) + the checkpoint fsync tail behind them (8–25 s, chronic)	every hour	≥ 8 alert-read kills 17:16–21:04Z Sep 21; three in the 19Z hour alone; every one a seconds-long transient	no band move — reader-side, landed
   D — the daily tier's single-transaction materialization (#3745)	query_store_stats_interval_daily writing 7.8–8.3 M rows in ONE transaction	once a day ~15:20Z	Sep 20: blew max_wal_size (1 GB) into a WAL-forced second checkpoint, four minutes of dead reads; Sep 21 under 16 GB: zero kills, timed checkpoints	storm class FIXED by the knob (#3802 authors it); slicing arm stays as the structural fix

   Why the bands do not have to be decided together after all

   The coupling risk was "moving one band's minute puts it on top of another". None of the four verdicts moves a minute:

   • A is fixed by a PERMUTATION inside the existing band, not a wider or shifted band. CompressionPhaseMinutes[index % Count] hands out minutes in registry order, so the three heaviest hypertables — registered consecutively — land on consecutive minutes and their 300–550 s runs overlap three-wide for ~5 minutes. The assignment becomes weight-aware: a declared, measured HeaviestHypertables set (today query_stats, query_snapshots, query_store_stats, with the measurement's date and the store class on the constant) takes the band's first minute and every Count / N-th minute after it (:35, :43, :51 for three in a 25-minute band), and the remaining tables fill the remaining slots in registry order. Same band width, same start, same HeaviestRefreshWindowMinutes — the 896 s ceiling and its 4 s margin are untouched, and RefreshCeilingProvenancePinTests does not move. What changes is which table starts on which minute; the pin is that no two of the heaviest set are within Count / N minutes of each other, plus the existing "no two policies contending for the same relation start on the same minute" invariant. A table joining the heaviest set is a measured decision, recorded on the constant.
   • B keeps its minute. The steady state (68 s) does not reach a 15 s MCP deadline; the staged first lifts are one-time and end ~Oct 12; the alert pass's reads are now retried once (The alert pass retries a store read once, two seconds later, before counting it failed — a write band's twelve-second stall no longer blinds an alert (#3848) #3851); the tick's own overview read moved to ≥ :37 (client side) and is memoized (get_fleet_overview: one collection-health scan per minute per host, however many overview calls race (#3735) #3738). The one remaining exposure — any other 7-day MCP scan issued at :35:00 — is a count, not a page. Close Aggregate compression starts at :35 on every store every hour — the same minute the fleet's readers read; a 7-day scan races a compression start by design #3781 as count-only, re-open bar: more than two losing hours a day after Oct 12, or any alert-read (not MCP) kill attributed to a B run. The coordination fact — the reader's minute and the writer's minute are the same by construction — goes on AggregateCompressionBandMinute as one comment line, riding A's PR (same file; fold-in policy).
   • C cannot move: it IS the hourly refresh, and its cost is the store's write volume plus the volume's fsync ceiling (25 s syncs during materialization hours are the disk talking — a gp3 IOPS/throughput bump is the hardware answer, the maintainer's ledger). Reader-side is the right side: The alert pass retries a store read once, two seconds later, before counting it failed — a write band's twelve-second stall no longer blinds an alert (#3848) #3851 landed (retry once, two seconds later, on a command timeout), V137's checkpointer row makes the sync tail visible on the store's own card from the next install, and The seven self-alert store reads in DarlingSelfAlertEvaluator still run without #3851's retry-once seam — route them through it as instance methods (follow-up to #3848) #3854 (pm-user-worker) extends the retry to the seven evaluator reads. Acceptance: the site-1 counter flattens after the 06:00Z install carries The alert pass retries a store read once, two seconds later, before counting it failed — a write band's twelve-second stall no longer blinds an alert (#3848) #3851.
   • D's storm class is fixed by the knob (out-of-band 16 GB live; The bundled store authors its memory knobs from the box but leaves max_wal_size at PostgreSQL's 1 GB default — derive it from disk headroom (≤16 GB) in the same versioned, heal-on-start block #3802 authors it from disk headroom on the next service-owned start). The slicing arm stays: an 8 M-row single transaction is a checkpoint bomb whichever max_wal_size it runs under, and the fsync tail scales with the burst — refresh_continuous_aggregate(view, start, end) over bounded bucket ranges (the RepairMaterializationHolesAsync shape, Three hourly rollups counted every restart's zero as a sample and a >24 h outage left a two-hour hole under a floor that said covered: interval-honest successors beside the legacy trio with the phase grid re-derived by its own method, and a targeted refresh of each materialization hole at service start (partial #3653 — items 16 + 9, Q12 + Q10) #3731) instead of one policy window. That is the lane Daily band re-derivation is DEFERRED until a daily successor is added (band full 23/23) — the slicing arm landed in #3865 (buckets_per_batch = 1); the storm class was fixed by the WAL ceiling (#3802) #3745 becomes. The daily band re-derivation the issue was filed for is DEFERRED until a daily successor is actually added (the band is full at 23/23 today; re-deriving it with nothing to add moves 23 jobs for no gain) — recorded on Daily band re-derivation is DEFERRED until a daily successor is added (band full 23/23) — the slicing arm landed in #3865 (buckets_per_batch = 1); the storm class was fixed by the WAL ceiling (#3802) #3745 as the sequencing rule.

   Lanes, in order

   1. The compression band puts the three heaviest hypertables on consecutive minutes: a 3.5-minute triple overlap starves alert reads nightly — the grid spaces by list position, not by measured runtime #3678 — weight-aware compression-phase assignment (permutation; measured heaviest set as a constant with lineage; pins). Rides the next nightly after tonight's; night-3 read is the acceptance (zero alert-read kills 00:36–00:59Z with the trio spread). Carries the Aggregate compression starts at :35 on every store every hour — the same minute the fleet's readers read; a 7-day scan races a compression start by design #3781 comment-line rider and closes Aggregate compression starts at :35 on every store every hour — the same minute the fleet's readers read; a 7-day scan races a compression start by design #3781 with the re-open bar.
   2. Daily band re-derivation is DEFERRED until a daily successor is added (band full 23/23) — the slicing arm landed in #3865 (buckets_per_batch = 1); the storm class was fixed by the WAL ceiling (#3802) #3745 — daily-refresh slicing (bounded bucket ranges per refresh call; WAL budget stated; the 15:20 → 15:33 cadence drift noted on the issue rides as a rider if it is a policy initial_start fact). Re-derivation deferred with the sequencing rule.
   3. C: nothing to lane; acceptance read after the install.

   Rulings none of this needs; product decisions stated so the maintainer can overrule: (a) B closes count-only; (b) D's re-derivation waits for a real daily successor.
3. erikdarlingdata

   added a commit that references this issue on Sep 22, 2026

   The compression band seats its three heaviest hypertables on spread m…

   ...

   27a204f
4. erikdarlingdata commented on Sep 22, 2026

   

   erikdarlingdata

   on Sep 22, 2026

   OwnerAuthor

   More actions

   Closed by the watcher: delivered in PR #3862, merged to dev.
5. erikdarlingdata

   closed this as completedon Sep 22, 2026