Converge each rollup to its SOURCE, not to raw (#1798)

[erikdarlingdata]

Closes #1798. Stacked on #1797 — based on its branch, so the diff shows only this change; GitHub retargets to dev automatically when #1797 merges and its branch deletes. Not based on dev because both touch RollupBackfill.

The defect

A fully-DONE --backfill-rollups left two hourly-tier retention policies HELD, against a contract line promising zero. The gate was right; the verb's completion claim was wrong.

Two notions of "covered" had drifted apart. The #1680 arming gate for an hourly-tier policy is source-relative — the daily must cover what the hourly holds — while the verb converged every rollup to raw's oldest row. All the dailies are hierarchical (time_bucket('1 day', bucket) FROM collect.*_hourly), so on a store whose raw purges are armed, raw is days deep while the hourlies hold weeks. A daily converged "to raw" stops short, the gate correctly refuses, and DONE prints over it.

The original #1759 profile hid this: on a held-purge store the hourlies are a subset of raw's span, so converging both tiers to raw satisfies the gate by construction.

The sharp edge is worse than the symptom. A hierarchical daily added after its hourly on a healthy store enters a hold nothing in the product can clear — the pre-raw region exists only in the hourly, and a verb aiming at raw never targets it.

The fix

RollupViews now carries each rollup's source and source time column alongside its raw table, and RollupBackfill.Targets is derived from that list rather than restating it — a second hand-kept copy is exactly the drift surface this issue is about.

RawTableFor stays raw for every rollup, dailies included. It answers a read question — where a window falls back to when this rollup cannot serve it — and raw remains the right answer. Only the backfill's convergence target moved.

The issue names three hierarchical dailies; there are four. query_stats_daily reads collect.query_stats_hourly too (verified in its DDL), and would have been left behind by a fix that took the list literally.

The harder half, found by the tests rather than by reading

The daily's plan was computed from a pre-backfill snapshot of its source. All plans are taken up front — necessary, since the disk preflight must total the whole job before committing to any of it — but a daily's source is backfilled earlier in the same loop. So the daily was aimed at where its hourly used to start and then judged against where the hourly now starts: SHORT on every daily, every time.

The run loop now re-probes and re-plans each rollup immediately before running it — the same measured-floor idiom that makes resume work, one level up. The up-front pass sizes the job; this one aims it.

I did not find this by reading. The verb's own end-to-end test went red, and I confirmed it was mine rather than pre-existing by re-running it on the parent commit, where it passes.

Contract

DONE now promises coverage of each rollup's own source, which is what each gate measures — made true rather than reworded.

Testing

Full solution rebuild: 0 warnings, 0 errors. All 37 rollup-backfill tests pass.

Behaviour pin: a hierarchical daily whose hourly out-reaches raw (raw purged to 4 days, hourly holding 12) must converge to the hourly's oldest bucket, and the hourly-tier policy must then arm. No-policy fixture — the ensure sweep's own refresh policy materializing contiguously already made a pin in this file vacuous twice.

Mutation (point the daily's source back at raw) verified RED on a compiling build, caught at two independent points — the source pin and the fixture-validity guard — each naming the cause.

One unrelated live test (DarlingAnomalyBaselineTests.EndToEnd_RestartExclusion…) fails on my local rig and fails identically on the parent with this change stashed, so it is pre-existing rig state. CI is the authority.

Field acceptance

Not run by me: the lab box is deliberately left in its 2-held state as the live reproduction, and the post-merge rehearsal there is the coordinator's.

🤖 Generated with Claude Code