Repository navigation
The hourly refresh envelope is stated at 306 s of slot margin; a measured run leaves 121.6 s and sits inside the warning band, and the daily series structurally cannot show it #3119
Description
HeaviestHourlyRefreshObservedCeilingSeconds documents its own invalidation condition — "what invalidates it is this number approaching RefreshPhaseStepMinutes x 60" — and that number has since been measured at 778.4 s, which is inside the warning band and leaves 121.6 s of slot margin where the envelope prose states 306.
This is not a claim that the constant is stale. Its doc deliberately CLOSES its population ("the scope has to CLOSE the population, not merely date it: 'up to 04:20Z' is a window that has ended and will still be true next year"), and that discipline is correct and worked. The 594 s figure remains true of the window it names. What has moved is the operating envelope the constant is stated against, which the same doc frames as a condition rather than a property.
The measurement
policy_refresh_continuous_aggregate query_store_stats_interval_hourly [1054], hourly self-metrics snapshots on 2026-09-06:
| snapshot | last_run_duration_ms |
total_runs |
|---|---|---|
| 20:13:19 | 675,557 | 347 |
| 21:13:23 | 778,423 | 348 |
| 22:13:28 | 665,928 | 349 |
| 23:13:34 | 705,218 | 350 |
total_runs advancing +1 per hour establishes run identity from the counter rather than by inference.
| seconds | margin against the 900 s slot | |
|---|---|---|
| doc ceiling | 594 | 306 s — "better than a third of the slot" |
| measured maximum | 778.4 | 121.6 s — 13.5 % of the slot |
RefreshSlotWarningSeconds |
750 | the measured maximum is 28.4 s INSIDE the warning band |
The doc's own threshold language: "A value at or past the slot width is asserted as a failure rather than accommodated." 778.4 s is not past 900 s, so this is not that failure — but it is 86.5 % of the way there, and ClassifyRefreshSlotHeadroom bands it Warning.
The part worth fixing regardless of the number: the surface cannot see maxima
get_store_metrics' daily series takes that day's LAST snapshot, not its maximum (DISTINCT ON ... metric_time DESC). For 2026-09-06 that is the 23:13:34 reading — 705.2 s.
| what you consult | 09-06 reading | apparent margin |
|---|---|---|
| the daily series (the obvious surface) | 705.2 s | 194.8 s |
| the actual hourly maximum | 778.4 s | 121.6 s |
So the standard surface overstates the margin by 73.2 s and, more importantly, places the day below the warning line when a run was 28.4 s above it. Anyone re-deriving this envelope from the daily series — the natural thing to do — concludes nothing entered the warning band.
The surface most likely to be consulted structurally under-reports exactly the quantity the envelope depends on. A ceiling validated against it will drift below reality undetectably, which is a property of the instrument and not of anyone's care.
This compounds a censoring effect already recorded on this route: the hourly snapshot is taken at a fixed offset, so a run longer than that offset is not observed at all in the hour it occurs, and an isolated over-ceiling spike is dropped rather than delayed.
Suggested direction
- Re-derive the envelope from a max-preserving source.
bgw_job_stat_historycarries per-run rows and is the route that can answer "what was the maximum", which the self-metrics series cannot. This is a read, not a code change, and should come first. - Update the envelope prose — the 306 s margin, and the "well inside its slot" reading — against whatever that source returns. Keep the closed-population discipline: state the window and close it.
- Note on the surface itself. Wherever the daily series is offered for this kind of question, say that each point is a last snapshot rather than a maximum.
get_store_metrics' description already carries careful warnings of this kind for other fields; this one is missing and it is the field an envelope check would reach for.
Step 1 is DONE — the maximum is confirmed on the uncensored route
bgw_job_stat_history was read directly. 2026-09-06's maximum is 778.4 s across 22 runs — so the figure is the day's true maximum, not merely the largest sampled, and the earlier caveat is discharged.
This also confirms the instrument property quantitatively: the daily series' 705.2 s understates the day by exactly 73.2 s, which is the figure predicted above from the last-snapshot mechanism alone.
The snapshot gap that covered 17:00–19:00 is filled: 473.1, 530.1, 523.1 s. Worth stating because it corrects a characterisation rather than just adding a point — 523.1 s is a FALL from 530.1 before the rise to 675.6, so the day's approach to its maximum is not the monotone climb it appeared to be from the incomplete series. Any argument resting on the length of that run needs re-deriving from the complete one.
So what remains here is step 2 (update the envelope prose against 778.4 s) and step 3 (make get_store_metrics say each daily point is a last snapshot rather than a maximum) — and step 3 is the one that stops this recurring.
Related
- RefreshSlotWarningSeconds rejects a 480s alternative on 'three of the five' readings — only one of the five listed readings reaches 480s, and that count is the sole stated reason #3107 made the 480 s rejection an ordering against this constant rather than a tally. That ordering is unaffected — 480 < 594 < 778.4 — so this does not reopen it.
A guard that stays GREEN while its claim stops discriminating
Three assertions in TimescaleSupportTests reference the ceiling. At a ceiling past 750 s, two go red and route correctly — their failure message already says to re-take the five-sixths decision rather than edit the assertion. The third, ceiling >= alternative, stays green and should not.
Its claim is the rejection argument for the 480 s alternative: "it would warn on the very run the compression grid is sized against." At a ceiling of 778.4 s that becomes true of the chosen 750 s line too — so the ordering stops separating the two lines. What expires is the argument, not the number, and nothing goes red to say so.
That is the failure mode where coverage looks intact because it passes. Whoever re-derives the envelope has to re-derive that argument as well, and only this note will tell them.
The blast radius is wider than that one test. Pre-existing assertions in both TimescaleSupportTests and RefreshCeilingProvenancePinTests key on the same ceiling-versus-watch-line relationship, and the doc's derived figures (the 306 s margin, the percentage above the ceiling) and the Verify population chain all move with it. Deliberately not stated as a count — the guards name each one, and a tally here would rot exactly like the one #3107 removed.
Activity
erikdarlingdata commentedon Sep 7, 2026
Claude posting for Erik Darling
Fixed by #3141, merged to
dev. Closed by hand.All three steps are done. Step 1 established the maximum on the uncensored route — 778.4 s across 22 runs on 2026-09-06, from
timescaledb_information.job_history— and confirmed the daily series understates that day by exactly the 73.2 s the last-snapshot mechanism predicted before anyone read the per-run route.The constant does not move, and the reasoning is the useful part. 594 is defined as the maximum of a published closed population, with two pins requiring that equality. The census that produced 778.4 s publishes a maximum and a run count, not 22 durations — so raising the constant would swap a re-derivable value for one a reader has to trust, which is the shape #3101 removed. Two arithmetic constraints agree independently: the constant is an
intand 778.4 is not, andCeiling * 10 % 60 == 0is required becauseCompressionPhaseMinutesquotes the ceiling in minutes to one decimal — 594 gives 0, 778 gives 40. So #3107's480 < 594 < 750ordering is untouched and the two assertions that red past 750 stay untripped.What changed is the envelope stated against it, now carrying its own closed window: 2026-09-06, 22 runs, maximum 778.4 s, 121.6 s of margin — 13.5 % of the slot, and 28.4 s PAST
RefreshSlotWarningSeconds, bandingApproachingSlot. The old margin sentence keeps its arithmetic but is attributed to the population it describes rather than to current load.Step 3 is the half that stops this recurring, and it is now in
get_store_metrics' own description:It names the per-run route, and records that the hourly snapshot carries the same limit one grain down — sampled at a fixed offset, so a run longer than that offset is never recorded at all. The pin asserts that sentence and the shipped SQL together, so a read that later became max-preserving reds the same pin rather than leaving the caveat wrong in the other direction.
Two operator-facing pointers were wrong in the same way and both are fixed rather than filed as the visible half:
HeaviestRefreshRuntimeSql's doc calledcollect.store_metrics"the complete per-run series", and theApproachingSlotlog line told an operator that series was the per-run history. So someone reacting to the warning band was being sent to a surface that structurally cannot show them the peak. Both now nametimescaledb_information.job_historyfirst, and the log line is pinned.Two further open-ended claims in the same file — beyond the three #3137 closed — are now scoped to the sets they were measured over.
Deliberately not recorded: the "guard stays green while its claim stops discriminating" note above. It expires only if the constant rises past 750, which it did not, and three routes now red for whoever raises it — including a new
Require(measured10 > Ceiling * 10)that fires the moment the ceiling reaches the measured maximum. Making the future failure mechanical is better than a note about it.OtherHourlyRefreshObservedCeilingSeconds' 140 s is still un-re-derived, and the ceiling's "ONE READING IS LEFT OPEN" paragraph is unchanged — one day's census discharges neither.