Skip to content

--backfill-rollups converges hierarchical dailies to raw's oldest, but the arming gate is source-relative: hourly-tier policies stay held on healthy stores after DONE #1798

Closed

Description

@erikdarlingdata

Observed (DARLING01 lab box, dev @ a0d83d7)

A full --backfill-rollups run completed and printed DONE, including the operator contract line promising that the next start reads N/N retention policies in place, N armed, 0 held paused pending backfill. The restart instead printed:

TimescaleDB: 16/16 retention policies in place, 14 armed, 2 held paused pending backfill

with these WARN lines naming the held pair:

Retention policy for procedure_stats_hourly created but HELD PAUSED - procedure_stats_daily does not yet cover everything it holds, so arming could drop history no rollup has.
Retention policy for query_stats_db_hourly created but HELD PAUSED - query_stats_db_daily does not yet cover everything it holds, so arming could drop history no rollup has.

The gate is right to refuse. The verb's completion claim is what's wrong.

Root cause

Two different notions of "covered" disagree:

  • TimescaleSupport.RollupViews pairs every rollup — including the hierarchical dailies — with its RAW hypertable, and the verb's convergence test (DarlingCliCommands, reached = finalFloor <= after.RawOldestUtc) converges each rollup to raw's oldest row.
  • The arming gate for an hourly-tier retention policy is source-relative: the daily must cover everything the hourly holds (daily min(bucket) <= hourly min(bucket)).

query_stats_db_daily, procedure_stats_daily, and query_store_stats_daily are hierarchical continuous aggregates (FROM collect.<x>_hourly, not raw). On a healthy store — raw purges armed, raw only a few days deep — the hourlies legitimately hold far more history than raw. Backfilling the dailies "to raw's oldest" leaves daily min(bucket) newer than hourly min(bucket), so the hourly-tier gates stay held even though the verb's own criterion ("every rollup now covers its raw table") is satisfied.

Why the original #1759 remediation profile doesn't hit this

On a store whose raw purges were held (the profile the verb was built for), the hourlies are a subset of raw's span. Converging both tiers to raw's oldest satisfies the hourly-on-daily gate by construction (day-floor <= hour-floor of the same instant), so 0 held is genuinely reached there. The gap only bites stores whose raw purges have already armed.

The sharp edge

Any hierarchical daily added after its hourly, on a store with armed raw purges, enters a hold that nothing in the product can clear: the daily cannot materialize the pre-raw region from raw (those rows are purged — the data exists only in the hourly), and the verb never targets the hourly as a floor. That's a permanently-held retention policy plus unbounded hourly growth, remediable only by a manual refresh_continuous_aggregate over the hourly's span. Same invisibility class as #1759, one tier up.

Fix sketch

Per-rollup source floor:

  • Carry each rollup's source relation alongside the raw mapping (hourlies -> raw table; hierarchical dailies -> their hourly view). RollupViews already exists as the single drift-proof list; extend it rather than adding a parallel one.
  • The verb's convergence target for a rollup becomes its source's oldest row/bucket; the completeness-by-construction invariant becomes floor <= source_oldest. Preflight, newest-first slicing, and the refuse paths are already per-view and carry over unchanged.
  • The DONE contract then genuinely implies 0 held on every store shape.
  • The read-side RetentionTierRouter mapping to raw stays exactly as-is — falling back to raw is correct for routing; this is purely the backfill verb's completion target.

Acceptance rig

DARLING01 is deliberately left in the 2-held state as the live reproduction: after the fix, a --backfill-rollups run there must extend the two dailies down to their hourlies' oldest buckets and the next restart must read 16/16 ... 0 held.

Activity

  1. erikdarlingdata commented on Jul 28, 2026

    @erikdarlingdata
    OwnerAuthor

    Correction from the fix work (PR #1799): there are FOUR hierarchical dailies, not three — query_stats_daily also reads FROM collect.query_stats_hourly (verified in its DDL; this issue's original grep pattern only matched two of the four FROM clauses, and inferred the wrong source for that view from the pattern's silence). A fix taking this issue's list literally would have left one rollup behind, still able to enter the unclearable hold.

    Second finding from the same work, not visible from reading: the verb plans ALL rollups up front (the preflight must total the job before committing to any of it), but a hierarchical daily's source is backfilled EARLIER IN THE SAME RUN — so the daily was aimed at where its hourly USED to start and then judged against where it now starts, reporting SHORT on every daily every time. The fix re-probes and re-plans each rollup immediately before running it (the same measured-floor idiom that makes resume work, applied one level up).

  2. added a commit that references this issue on Jul 28, 2026
  3. erikdarlingdata commented on Jul 28, 2026

    @erikdarlingdata
    OwnerAuthor

    Fixed by PR #1799 (merged to dev at 18c12c7): convergence is SOURCE-relative — RollupViews carries each rollup's source relation and time column, the backfill targets are derived from that one list, the preflight bounds hierarchical dailies via EventualSourceFloor (the deeper of the source's current floor and raw's oldest), the run loop re-plans each rollup from live floors immediately before running it, and a mid-run refusal is reported as a refusal rather than success.

    Field-accepted on the lab box the same day, exactly per this issue's acceptance-rig plan: the deploy start printed the two expected HELD PAUSED warnings, the dry run planned precisely the two source-relative gaps the raw-relative verb had called covered (procedure_stats_daily back to 2026-07-20, query_stats_db_daily back to 2026-07-22 — each to its hourly's oldest bucket), the real run COVERED both, and the post-restart startup line reads:

    TimescaleDB: 16/16 retention policies in place, 16 armed, 0 held paused pending backfill
    

    DONE now genuinely implies zero held on every store shape.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions