Repository navigation
Watch the heaviest hourly refresh's runtime against the 900-second slot it has to fit in #3044
Description
#3041 pinned the compression policies to a phase grid whose sizing rests on an explicitly stated operating envelope, asserted at build time as:
HeaviestHourlyRefreshObservedCeilingSeconds < RefreshPhaseStepMinutes * 60
That is the right shape — a conditional downgrade written only in prose reads as unconditional to whoever finds it next. But the assertion is a build-time bound against a recorded ceiling, and the thing it bounds is a runtime that moves with data volume. Nothing today watches the live figure approach the bound; the assertion fires only if someone changes a constant, not if reality changes underneath the constant.
The measurement
The heaviest hourly continuous-aggregate refresh, against the 900-second slot width the grid divides the hour into:
| reading | seconds | % of the 900 s slot |
|---|---|---|
| pre-narrowing band | 3,301–6,330 | (window was 7 days) |
| ceiling under the 1-day window | 864 | 96% |
| #3041's own recent snapshot | 194 / 225 / 335 | 22–37% |
| observed since | 593 | 66% |
| settled after | 465 | 52% |
The 194–335 s readings are what #3041's narrative was written against, and they read as comfortable. The 593 s reading was taken after that snapshot, and 864 s was already recorded earlier under the same window — so the honest characterisation is not "comfortable, occasionally spiking" but a runtime that has been observed anywhere between 22% and 96% of its slot, trending with volume rather than settling.
Why this is its own issue rather than a note on #3035
#3041's grid survives the heaviest refresh running long, and by a wide margin: it would have to exceed 1,320 s to reach the earliest compression minute and 3,540 s to reach the latest. That is not the exposure here.
The exposure is the slot itself. At 900 s the refresh runs into its own neighbour in the refresh family, which is #3024's grid, not #3041's. Past that point the answer is not a different phase for compression — it is that a 15-minute grid can no longer hold thirteen hourly refreshes, and the fix is a redesign (fewer slots, a longer cadence for the heavy aggregate, or splitting it) rather than a renumbering. Filing it against #3035 would attach a refresh-family capacity question to a compression-phase change that is already merged and correct.
What would make this observable
The assertion covers the constant. What is missing is a signal on the live figure crossing a fraction of it — the natural home is wherever job runtimes are already read, since the converge path reads timescaledb_information.jobs and the store already keeps job history. A threshold well below 900 s (the 66% reading would already have tripped a 60% line) turns "someone notices during an unrelated investigation" into a thing the tool says.
Worth deciding deliberately whether it belongs as a band, an alert, or simply a logged warning on the converge path. A band implies an operator action, and the action here is a scheduling redesign rather than anything immediate — which argues for the quieter form.
Provenance
The 593 s and 465 s figures came from a live check against the store's own job history during #3041's review, not from the PR's fixtures. The pre-existing 864 s ceiling and the 194/225/335 s readings are #3041's own, cited there.
Activity
erikdarlingdata commentedon Sep 5, 2026
Claude posting for Erik Darling
The watch criterion should come from the code rather than from anyone's judgement, and #3041 as merged supplies it. Updating this issue with numbers verified against
origin/devrather than restated.TimescaleSupport.cs:1766—HeaviestHourlyRefreshObservedCeilingSeconds = 864.TimescaleSupport.cs:1655—RefreshPhaseStepMinutes = 15, so the slot is 900 s.The margin the merged code already lives on is 36 seconds. 864 against 900 is 96.0% of the slot. That is not "an envelope with room in it"; it is an envelope whose recorded ceiling is four percent short of invalidating itself.
And the assertion says what happens past it rather than leaving it to be inferred — it fails with "it no longer fits inside its own slot, so excluding one slot is no longer enough and the compression grid has to be re-derived." So the code's own position is that crossing 900 s is a re-derivation, not a tuning.
The criterion, taken from that rather than invented
Tonight, against those lines
Closest approach tonight was 306 seconds clear, on a rising-under-load trajectory — which is the trajectory the envelope is written about. Below the 864 s post-fix baseline throughout, so this is not the pre-fix curve returning, and 594 s taken alone invites exactly that misreading.
The instrument is a chart maintained by the fable-loop session, and it is currently the only thing watching this. That is the gap this issue was filed on and it has not closed — the build-time assertion covers the constant, nothing covers the live figure approaching it. What has changed is that the threshold no longer needs choosing: 750 and 900 are derivable from the code.
Provenance
The 594 s and 465 s figures and the 750/900 criterion are fable-loop's, from the live store's job history during #3041's review. The constants and the assertion text are verified here against
origin/devatTimescaleSupport.cs:1655and:1766. The 36-second margin is arithmetic on those two.erikdarlingdata commentedon Sep 5, 2026
Claude posting for Erik Darling
Update: the envelope is receding, and it held through the worst of the day. Readings verified against the 900-second slot rather than restated.
594 s was the peak and it is now two readings behind. Friday-evening load is draining and the runtime with it, so the compression grid is moving further from its stated invalidation condition rather than toward it. Worst margin all evening was 306 seconds clear; against the 750 s warning line (83.3% of slot) the peak reached 66.0%. No warning, nowhere near invalid.
Worth recording why this reading carries more weight than the ones this issue was filed on: it is the first time the instrument has been read against merged code's own declared assumption rather than against anybody's judgement. The 750 and 900 thresholds come from
HeaviestHourlyRefreshObservedCeilingSecondsandRefreshPhaseStepMinutes * 60, so "receding" is a statement about the code's envelope, not an opinion about a trend.The boundary-chunk collision now has a forecast: ~02:48Z, plus or minus a cycle
The drift is exactly finish-to-start, confirmed to the second: finished 20:18:17 → started 21:18:18, finished 21:24:17 → starts 22:24:17. So the next start is the previous finish plus one hour, and the start-minute therefore advances by the runtime itself each cycle — about 6 minutes now, down from ~10 at the 594 s peak.
Progression from here:
:24 → :30 → :36 → :42 → :48 → :54, against compression job 1029 pinned at :52:29 and running 13–21 s. The:48start is the candidate, because its ~6-minute window is the one that contains:52— verified: a:48:00start at 359 s covers through:53:59.And the irony is worth having in writing, because it makes a quiet night an informative outcome rather than a wasted one. Falling runtime both delays the test — smaller steps mean more cycles to walk to
:48— and narrows the window it would test, since the eligible-and-inside-the-window set is only as wide as the running refresh's age. At 594 s that window was 9.9 minutes; at 359 s it is 6.0. So a quiet night may let 1054 step past:52untouched. That is a real result, not an inconclusive one: it would mean the boundary-chunk exposure is bounded by load, which is exactly what #3041 argued and what this issue exists to check.Provenance
Readings and the forecast are fable-loop's, from the live store's job history. The percentages, margins, the finish-to-start deltas, the step arithmetic and the
:48window containment are recomputed here from those figures against the constants atTimescaleSupport.cs:1655and:1766.erikdarlingdata commentedon Sep 5, 2026
Claude posting for Erik Darling
Correcting the forecast in my previous comment: ~02:48Z is wrong, it slips to ~03:49Z — and the reason is the effect that comment flagged, now observed twice.
Fifth reading, and the envelope keeps receding:
Monotonic decline from the 594 s peak — now under a third of the slot. The grid is comfortably inside its stated limit and moving further inside it.
Why the forecast moved. The walk is confirmed finish-to-start again to the second (finished 22:29:10 → next start 23:29:10), and each cycle advances the start-minute by the runtime. So a shorter run advances it more slowly: the step is now ~4m53s, down from ~10 minutes at the peak. Projected starts
23:29 → 00:34 → 01:39 → 02:44 → 03:49 → 04:53.Recomputed against compression 1029 at
:52:42: the 02:44 window (02:44–02:48:53) misses:52, and the 03:49 window (03:49–03:53:53) contains it. So ~03:49Z, and it is marginal — 1029 is itself creeping ~7–20 s later per cycle by its own runtime, so by then it sits near:53, close to the window's trailing edge.This is the second slip from the same cause, which makes it a property rather than a coincidence: falling runtime pushes the test later and shrinks the sliver it would test. A quiet night may let 1054 step past
:52without ever overlapping. That remains a result and not a miss — a near-miss with the geometry recorded answers this issue's margin question as usefully as a hit would, because the width of the exposure is exactly the running refresh's age.Readings and projection are fable-loop's; the percentages, headroom, step sizes and both window-containment checks are recomputed here.
Fixes #3044 (#3055)
erikdarlingdata commentedon Sep 6, 2026
Claude posting for Erik Darling
Closed by #3055, merged to
devat7205cd37fa201469c4508edd4b0ac2233d83bf58. Closing by hand because a closing keyword does not fire againstdev.The signal is a logged warning, and the decisive argument is not the one this issue anticipated. It is not "the action is a scheduling redesign, so the quiet form fits." It is that a second alert at the same threshold would be redundant without being reliable: #2136's Store Job Over Cadence already judges this job's
last_run_duration, and its 25% default lands on exactly 900 s — one slot — because a slot is3600 / RefreshPhaseSlots.Thresholds are derived rather than chosen:
RefreshPhaseSlotSeconds = RefreshPhaseStepMinutes * 60, andRefreshSlotWarningSeconds = slot * 5 / 6, exact at every integer step. So changing the step moves the warning line with it, and a mutation freezing one literal is caught by the cross-multiplied identity rather than by either literal alone.The best detail is one this issue did not ask for.
SlotExceededstarts at>=the slot width because the build-time assertion isceiling < step * 60— so the runtime watch and the constant's own assertion agree on where the wall is by construction, not because two comparisons happen to match.And the level at the wall is Error, not a warning or an alert, because crossing 900 s means the grid must be re-derived rather than tuned. That is not a warning about a future state. Routine readings log at Debug, since an hourly Information line about a healthy job is how signal gets buried.
Storage: none. No rung, no table, no collector, no timer.
collect.store_metrics(object_kind = 'background_job', V56) already storeslast_run_duration_msandschedule_interval_msper job per hour. What was missing was the bound, not the history.A correction to my own routing. I sent the work to
ConvergeCompressionScheduleAsync, which was wrong three ways: that read is scopedproc_name LIKE '%compression%'and the heaviest refresh is apolicy_refresh_continuous_aggregatejob absent from it;last_run_durationlives injob_statsrather thanjobs; and the functionalter_jobs every stale row it reads, so a refresh job appearing in its list would have been given the compression cadence and a compression phase minute. Both converges also run once per service start, so a month of uptime would have logged the figure once. The hourlyEvaluateCompressionJobHealthAsyncwas the right home.This retires a manual watch. The 1054 runtime chart has been carried by hand every hour since 20:13Z; the product now reports it.
Not verified: no real
last_run_durationreading was produced — foregroundrun_jobdoes not update that accounting, so the live test asserts SQL resolution, three-way join reach, and that a pre-run read is an honest empty, with the logger staying silent as the assertion. TheDarlingWorkersweep wiring is source-reviewed rather than driven.#3060 carries the occupant's own defects, which this PR documented while establishing that a second alert was the wrong shape.
Fixes #3060 (#3066)