fix(replication): full-copy when a source has no resume cursor (#426)

[kriszyp]

Fixes the confirmed, common data-loss in #426. Opened as a draft / interim because cross-model review surfaced a foundational interaction (below) that a proper fix must also address — I'm taking that on as a follow-up.

The bug (#426)

A follower receiving a source's writes relayed through a bridge (a proxied / non-leader subscription) has no top-level resume cursor for that source. On reconnect the subscription build fell into the "starting from scratch" branch, which resumed a non-leader source from Date.now() - 60000 — silently claiming the follower already holds everything older than a minute. When the follower is actually behind on records now older than that window (it missed them while the bridge was down), they are never re-requested and never delivered — permanent, silent divergence even though the follower reconnects cleanly and reports connected: true.

Reproduced deterministically and (originally) via the rapid-reconnect-adversity stress test ([500,499,304], 39% drift).

The fix

replication/replicationConnection.ts: when no resume cursor resolves (startTime === 1), request a full copy (startTime = 0) instead of now - 60000. A full copy is idempotent and the leader collapses redundant requests via the per-connection min(startTime); an occasional extra copy is acceptable versus silent data loss.

Tests

• New: integrationTests/cluster/proxiedResumeBacklog.test.mjs — deterministic regression. A follower misses a batch while its bridge is down, the batch ages past the 60s window, the bridge returns, and the follower must still converge. Fails before this change, passes after.
• Instrumentation: the rapid-reconnect-adversity stress test (the original detector) now logs per-poll convergence, the last-victim index, and an on-divergence cluster_status dump so a future failure is diagnosable.
• All related replication tests pass: proxiedLeadingDuplicateSkip, leadingDuplicateSkip, addNodeFullCopy.

⚠️ Known limitation — why this is a draft

Requesting a full copy for one cursorless source collapses the whole WebSocket to a base-copy via the leader's currentSequenceId = min(startTime). Base-copy is additive — it ships only the leader's live primaryStore rows and resumes audit replay from copyStartTime — so it does not reconcile deletes:

• A co-subscribed source that had a valid cursor but missed a delete before copyStartTime can be left with a stale record (Codex flagged this; reachable only in multi-path mesh topologies, not the common bridge case — I could not construct a deterministic repro).
• A cursorless source's own pre-copy deletes onto partial stale data are likewise not reconciled.

Both are refinements of a pre-existing base-copy property (base-copy onto a non-empty follower never reconciled deletes — already true for fresh / predates-retention copies). This change does not introduce that property, but it does exercise it more often. So this fix is a net improvement for the common insert-loss case, not a strict improvement for delete-heavy workloads in rare topologies.

Follow-up (in progress): the proper fix — per-source post-copy resume (don't collapse a cursored co-subscribed source to copyStartTime) and/or base-copy delete-reconciliation. Tracked on #426.

Closes nothing yet — keep #426 open for the foundational follow-up.

🤖 Generated with Claude Code

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[claude]

Reviewed; no blockers found.

[kriszyp]

Scope correction — keep open, but this does not fix the #426 stress failures

Follow-up to the verification work on #426. Re-labeling this PR's scope based on an empirical result:

I built this branch into a local dist and ran backlog-recovery (CI-faithful OFFLINE=30 / KEYS=800) with the fix applied — it still froze ({"A":800,"B":350,"C":800,"D":800}, gap 450, first run). That failure is bug (b) — the send-side stale-cursor freeze (leader never re-streams [T, head]) — not the cursorless → now-60s path this PR changes. Details + root-cause lead in #426.

So the accurate framing for this PR:

• ✅ It is a correct fix for the cursorless resume decision (startTime === 1 → full copy instead of now-60s), and it's idempotent/harmless.
• ⚠️ But we currently have no reproduction where it changes the data outcome. The new proxiedResumeBacklog cluster test self-heals via the follower's direct cursor, so it only exercises the resume-decision log path (corrected the test's claims in 36ecd92); and backlog-recovery — which the original description cited as the repro ([500,499,304]) — is actually bug (b), unaffected by this change.
• The body's "Fixes the confirmed, common data-loss in rapid-reconnect stress: follower permanently loses replication backlog after restart churn (data loss; fast-skip ruled out) #426" overstates it. Suggest softening to "hardens the cursorless resume decision; does not address the rapid-reconnect stress: follower permanently loses replication backlog after restart churn (data loss; fast-skip ruled out) #426 send-side freeze (bug b), tracked separately."

Recommend keeping it open as a correct-but-narrow hardening, decoupled from the #426 stress failures. No code change requested here — just aligning the claims with what we can show.

— Claude (Opus 4.8)