Skip to content

Aggregate compression starts at :35 on every store every hour — the same minute the fleet's readers read; a 7-day scan races a compression start by design #3781

Closed

Description

@erikdarlingdata

The aggregate compression band starts on the minute the fleet's readers read

#3611 gave the twenty-three continuous aggregates a once-a-day compression each, one aggregate per hour at registry position, all at minute AggregateCompressionBandMinute = HeaviestRefreshStartMinute + HeaviestRefreshWindowMinutes − 1 — :35 (TimescaleSupport.cs:6074). The derivation is sound on its own terms: the minute is the last one of the heaviest refresh's window, which the hourly grid holds and does not use, and joining #3174's raw compression band would have widened it and eaten the 896 s ceiling (the doc paragraph above the constant says exactly why).

What the derivation did not weigh is that :35 is the minute the monitoring seat's hourly tick reads every store — and, more generally, the minute a human or an agent asking "how is the fleet" at the top-of-hour-ish lands on. On the first day the band was live (2026-09-19/20, .453), the get_fleet_overview collection-health scan on the largest store was cancelled by the mcp role's 15 s statement_timeout at 06:35, 11:35, 12:35 (before #3738 removed the caller's ×3 fan-out) and again at 00:35 and 03:35 (after it — single scan, no fan-out). Each hour a different aggregate's daily compression starts at :35:00; the hours that lose are the ones whose aggregate had a large enough day of fresh chunks to stall a 7-day scan past 15 s. Retries at :36+ always succeed.

Why this is structural, not night-0

The night-0 staging (StageAggregateCompressionNights) governs only each aggregate's FIRST heavy run. Steady state is twenty-three daily runs, one per hour 01–23Z, every one starting at :35:00 on every store. The reader's minute and the writer's minute are the same by construction, for the life of the grid.

Options, smallest first (the #3678 shape)

  1. Move the aggregate band minute off :35 — one constant. The hourly grid's free minutes are known (CompressionPhaseMinutes derivation, guard windows); the doc paragraph at :6066 already argues the band must not join Re-derive the refresh grid so contention does not depend on list position: no step works, because Other*4 < Heavy has no step term #3174's raw band, so the candidate is another unused minute of the heaviest window, or the first minute after the raw band closes (:59 → wraps to :00 of the next hour, which collides with the light phase — the worker measures). Whatever minute is chosen, the tick charter's read minute must not be the same one — that is a coordination fact to record beside the constant.
  2. Accept count-only. Retries land; the impact is one cancelled read per losing hour per store. Cheap, and honest if the losing-hour count stays ≤ 2/day after night 2.

The monitor seat's job_history read (queued behind SSO) names hour 03's aggregate and its 03:35 duration tonight; the night-2 band read for #3678 (00:36–00:59Z) is the same evening. Decide this WITH #3678: both are "one constant moves a band off a minute readers use", and moving one without the other can put them on top of each other.

Client-side mitigation is already in the tick charter (the overview read sequenced last, ≥ :37); this issue is the product half.

Activity

  1. erikdarlingdata commented on Sep 20, 2026

    @erikdarlingdata
    OwnerAuthor

    Confirming datum from the store (job_history, owner role): the 03:35Z kill's writer was job 1109 — query_store_stats_hourly's aggregate compression — started 03:35:00 exactly, ran 1,246 s. That was its staged night-1 lift at its registry hour (the first heavy run the night-0/1 staging governs); the same aggregate's steady-state daily run is 68 s-class.

    Sizing consequence: the band-minute collision bites hardest during the staged first lifts (twenty-three of them, one per night, largest first — so the next ~3 weeks of nights each carry one) and thereafter only in hours whose aggregate had a heavy day. With #3738's memo (a retry within 60 s is served from memory) and the tick's ≥:37 sequencing, the remaining exposure is any other MCP consumer reading a 7-day scan at :35 sharp.

    Instrument 4 in the same batch: no refresh-family job over 92 s since 20Z (#3731's grid holds ~10× under the 896 s line).

    Decision still coupled to #3678's night-2 read (2026-09-21 00:36–00:59Z). Night-1 record for that: the raw trio ran full weight (229 / 255 / 490 s, ~4 min three-wide overlap) and produced zero alert-read kills.

  2. erikdarlingdata commented on Sep 22, 2026

    @erikdarlingdata
    OwnerAuthor

    Decided WITH #3678 (design on that face, comment above): count-only, with a re-open bar — steady-state runs are 68 s-class; the killer runs were the staged first lifts, one per night, ending ~Oct 12; the alert pass retries once (#3851); the tick moved off :35 and is memoized (#3738). The reader-minute coordination fact goes on AggregateCompressionBandMinute as one comment line riding #3678's PR, which closes this issue. Re-open if: more than two losing hours a day after Oct 12, or any ALERT-read (not MCP) kill attributed to an aggregate-compression run.

  3. added a commit that references this issue on Sep 22, 2026
    27a204f
  4. erikdarlingdata commented on Sep 22, 2026

    @erikdarlingdata
    OwnerAuthor

    Closed by the watcher: delivered in PR #3862, merged to dev.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions