Repository navigation
Replication W2: Cursor correctness & divergence detection #432
Description
Activity
Suggested acceptance test: a deterministic 4-node "cursorless relayed-origin" divergence repro
While investigating #426 I confirmed a gap worth capturing as a W2 deliverable: we currently have no deterministic test that reproduces the silent
[T, head]gap. The existing regression (integrationTests/cluster/proxiedResumeBacklog.test.mjs) does not — I verified empirically that in its 3-node L→B→M line the follower tracks a real direct cursor to its bridge and re-acquires the backlog regardless of the resume decision, so it self-heals and convergence passes even on the buggy path. (I've corrected that test's misleading claims on the #428 branch — commit 36ecd92 — to label it a resume-decision/smoke check, not a data-loss guard.)The only thing that reproduces real loss today is the 4-node
backlogRecoverystress test, and only intermittently (it depends on hub-emergence under churn). CI run 27897883840 caught it cleanly: node B frozen at{"A":800,"B":553,"C":800,"D":800}, with B holding a real cursor for peers A/C butstart time: 1(no cursor) for peer D — whose writes had reached B only relayed.Proposed W2 acceptance test
A deterministic cluster test that forces the genuine condition, to serve as the gate for divergence-detection (and as a regression for #428):
- ≥4 nodes where one origin's writes reach a target node only via a relay (no direct data txn on the target's direct connection to that origin), so the target's per-source cursor for that origin stays at
1. - The target is offline across a write window that ages past the non-leader resume floor (
now - 60000), then returns with no co-subscription path that re-delivers the aged backlog (the masking path that makes the 3-node test self-heal). - Assert record-count convergence (real data), not a log line.
The hard part is making "this origin is known only via relay" deterministic rather than waiting for emergent hub formation under churn — likely by shaping
add_nodele/relay topology so the target never opens a direct data subscription to that origin. If we can pin that, it becomes the canonical red-before/green-after test for:- W2 here — receive-side sequence-gap detection must catch and heal the
[T, head]hole regardless of how the cursor got stuck. - W4 (Replication W4: Per-origin transaction-log convergence (keystone) #434) — per-origin cursors should make the
start time: 1state unreachable in the first place (the relayed origin's applied position is tracked, not the proxy's).
Flagging it here so the divergence-detection work lands with a deterministic harness instead of leaning on the intermittent stress job. Happy to take a crack at the topology-shaping once the foundations are scoped.
🤖 Filed by Claude on behalf of Kris.
- ≥4 nodes where one origin's writes reach a target node only via a relay (no direct data txn on the target's direct connection to that origin), so the target's per-source cursor for that origin stays at
Scoping note: the #426 silent-gap that motivates this workstream is root-caused to HarperFast/rocksdb-js#668 (a committed-watermark pin from an
IsBusy-retry orphaned position), fixed upstream in the storage engine. W2's receive-side gap-detection is still the right durable safety-net — it would have caught this class regardless of cause — but the specific #426 data loss is an engine bug, not a replication-protocol one.- added a commit that references this issue
on Sep 2, 2026 - added a commit that references this issue
on Sep 2, 2026 - added a commit that references this issue
on Sep 2, 2026 - added a commit that references this issue
on Sep 3, 2026 - added a commit that references this issue
on Sep 3, 2026 - added a commit that references this issue
on Sep 3, 2026 - added a commit that references this issue
on Sep 9, 2026
Metadata
Metadata
Assignees
Type
Fields
Priority
Workstream W2 of #430 · Foundation · the silent-data-loss safety net
Status update (2026-07-01)
Design direction 1 (explicit error classification) is now substantially shipped as point fixes, and the current-state text below is stale on the details:
isPermanentSourceBlobErrorCodeis no longer ENOENT-only — it'sENOENT || 404 || 500, with the origin forwarding a structured statusCode (Incomplete (truncated) source blob is misclassified as transient → permanently wedges replication cursor (ENOENT is handled; 'Blob is incomplete' is not) #429 → fix(replication): classify gone/corrupt source blobs as permanent via forwarded statusCode (#429) #443, plus Blob replication wedges permanently when source blob is gone (ENOENT) on an expiration cache table — held resume cursor never recovers #403 → fix(replication): advance resume cursor past source-missing (ENOENT) blobs instead of wedging (#403) #405 for the original ENOENT advance-past).blobRepair.ts).What remains is exactly the two items this workstream predicted, and both have live field evidence:
blobRepair.ts) is the repair loop that makes advance-past safe.Summary
Convergence rests on a single per-(db,node) resume cursor whose advance is gated by hand-rolled, narrow error classification. It's a trusted watermark, not a checked one, and it breaks two opposite ways — over-holding (wedge forever) and silent gaps (undetected data loss). There is no receive-side sequence-gap detection, which is the single highest-value missing safety net in replication.
Root cause / current state (see status update for what has since shipped)
T, but the leader's per-subscriber send-position diverges and only re-sends the head, so the[T, head]window is never re-sent. Nothing detects the hole because the live tail keeps arriving andconnected:trueholds (rapid-reconnect stress: follower permanently loses replication backlog after restart churn (data loss; fast-skip ruled out) #426 — observed as ~39% silent data loss).Design direction
.code; classification is a table with a safe default plus a bounded-retry budget. An error that holds the cursor is counted, escalated after N reconnects, and after a bounded number of identical-error reconnects advances-and-marks-diverged rather than holding forever. (Largely shipped — the remaining piece is the bounded-retry budget for the transient class.)[T, head]hole is detected and surfaced (feeds the W8 divergence alarm) instead of masked by live tail traffic.blobRepair.ts.) Extend it to close the loop behind any new advance-past classes from item 1.Scope
.code/statusCode on blob error paths + broadened permanent classification (fix(replication): classify gone/corrupt source blobs as permanent via forwarded statusCode (#429) #443, harper#1480)blobRepair.ts)Retires / advances
Dependencies
Pairs naturally with W1 (which provides the surfacing channel) and W8 (which raises the alarm). The send-side residual pairs with W4.
Effort / risk
M / medium (down from L — classification and backfill shipped; the budget + gap detection remain).
Acceptance criteria
[T, head]gap raises a detectable signal rather than silently persisting.🤖 Filed by Claude on behalf of Kris.