Proactive blob backfill: repair already-committed records whose blobs are missing/corrupt (no recovery once the resume cursor advances past them) #388

Gap

Split out of #386 (self-repair half). #368 added the async durability watermark so the resume cursor holds on an in-flight blob gap and a reconnect re-streams the disrupted blob — this prevents new loss and recovers gaps within the in-flight window. But there is no mechanism to repair a blob that has already diverged after its record committed durably and the cursor advanced past it.

Confirmed by tracing the current code:

• No read-time repair. When a blob read hits Incomplete blob (core/resources/blob.ts:189, :1167), the error just propagates — there is no re-request of that blob from a replication peer. (Cache re-source via sourcedFrom exists for caching tables, but that is application-specific, not blob-repair infrastructure, and authoritative tables have no source to fall back to.)
• No reconciliation sweep. Nothing periodically checks for blob-level divergence across peers and heals it. The only blob sweep is orphan cleanup (core/resources/blob.ts:~1281), which deletes unreferenced files — the opposite of repair.
• fix(replication): watermark-based blob-gap handling (no data loss + no deadlock) #368's recovery is in-flight-only. replicationBlobRepairAuthoritative.test.mjs exercises exactly the held-cursor + reconnect-restream path; it does not cover already-committed divergence.

Net: if a blob file is missing/corrupt after its record committed and the cursor moved past it (the [customer-cluster] shape in #386, where records committed but the bytes are absent on every node), there is no proactive recovery path. On an authoritative table the read fails permanently.

Ask

A blob-level reconciliation / backfill path that:

1. Detects records whose referenced blob is missing/incomplete/corrupt on this node (independent of the in-flight watermark window). The new cluster_status blob-divergence signal (feat(replication): surface blob-replication divergence in cluster_status (#386) #387) gives the observability hook; this is the repair side.
2. Re-requests the missing/corrupt blob from a peer that has a complete copy (extend the existing per-record fetch / getRecord path to a blob-targeted request), and
3. ideally a bounded reconciliation pass that can run on reconnect or on demand to converge a node that diverged outside the audit-retention/base-copy window.

Related

• Sustained blob-replication timeouts cause silent, unrecoverable blob loss with no operator signal or self-repair ([customer-cluster]: 344/345 attachments, ~663 MB) #386 — umbrella (sustained blob-replication divergence → silent unrecoverable loss). Observability half shipped in feat(replication): surface blob-replication divergence in cluster_status (#386) #387.
• fix(replication): watermark-based blob-gap handling (no data loss + no deadlock) #368 — async durability watermark (in-flight gap hold + reconnect re-stream). Prevents new loss; does not backfill already-diverged blobs.
• Blob repair sweep has no automatic trigger — a materialized corrupt stub needs an operator to run repair_blob_data #385 — interrupted receive leaves a corrupt stub and never re-requests it (the per-blob re-request primitive this would build on).
• Replication: force bounded base-copy resync when a peer is behind > audit retention #277 — forced bounded base-copy resync (record-level; does not reconcile blobs whose records already committed).