Resume a removed or relayed origin from a per-origin cursor

[kriszyp]

⊙ Problem

After remove_node, every resubscribe on every surviving node asked its peer for the removed node's entire retained transaction log, starting at 0. That covers restarts, deploys and reconnects. Every node added later pulled the same log from 0 in its base copy's tail, then relayed it back out. Any origin a subscription neither lists nor excludes has the same gap: an origin that reaches the subscriber only through a relay has no cursor either.

Idle-peer resume remains deferred to harper-pro#922, the residual of #989 defect B. A reconnect past auditRetention follows the existing bounded base-copy resync. If an idle sender still retains a last local key older than that cutoff, it can copy again on every reconnect; an empty local log uses the copy's wall-clock anchor and reaches the cutoff after another retention window.

❓ Your call: Is this the right scope? Yes: this PR addresses removed and relayed origins; idle-peer resume belongs to #922. It is W4 (harper-pro#434) scope item 1, the per-origin cursor vector fed into startByLog, shaped so W4 extends it. A removed member's earlier writes stay valid, so nothing deletes a log (harper-pro#686).

💡 Solution

Every origin now has a resume position on every connection, behind a new originCursors capability. RocksDB builds advertise it; a peer without it does not exchange cursor vectors or apply incremental cursor floors. Exact relayed-log anchors also bound base-copy tails sent to older peers.

Where	Before	After
Restart of a survivor	Peer resends the removed origin's whole retained log	Peer starts that origin 60 s below the survivor's cursor for it
Origin reaching a node only by relay	Whole retained log on every resubscribe	Same per-origin start
Base copy to a new node	Copy tail pulls every relayed log from 0	Tail resumes each relayed log in append order past the key it held at the copy anchor; the new node keeps those keys as cursors. A relayed log whose boundary cannot form starts from 0, as before

The pieces:

• Recording (receiver). The receiver records, per connection, the highest origin log key it has durably applied for every origin, in the [seq, peer] row's nodes[].originLogKey. It is held under the same blob rule as the durable watermark, and core merges it (companion core PR).
• Sending (receiver). The receiver sends those cursors on SUBSCRIPTION_REQUEST[4], and fresh ones with an includeNodes update. A removed origin is seeded from its own connection's row, when its hdb_nodes row is deleted or physically absent.
• Applying (sender). The sender applies cursor − [ORIGIN_CURSOR_OVERLAP_MS](https://github.com/HarperFast/harper-pro/pull/998/changes?w=1#diff-e707e24c4bc1646c60ac7da292c9707439ee0083f363f0433fda2f851c2b20d1R2748) to each unlisted, unexcluded origin (resolveOriginFloors), in the incremental range's startByLog, in the subscription predicate, and on the includeNodes path. An origin with no cursor still starts at 0.

⚖️ Alternatives

❓ Your call: The original planning review (codex) returned Framing-Verdict: better-alternative-exists. Its exact append-order anchors for base copies and per-origin seeding cursor were adopted. The owner chose the 60 s overlap for incremental resume (Q2 a); #922's closed floor replaces that residual. The follow-up scope removal is owner-directed and both its initial planning review and required post-review framing recheck returned Framing-Verdict: chosen-approach-sound.

Approach	Why it lost
List removed and relayed origins as ordinary nodeSubscriptions entries	A listed proxied node resumes from nodes[].seqId, which is the relay's local time, not the origin's key; main deliberately does not use it as a resume bound because it can sit ahead of what was applied. It would also make a removed node a subscription member again
Exclude a removed origin from every subscription	Its last writes that had not reached every node would never arrive
Delete or purge the removed origin's log	Ruled out on #686: the writes stay valid
Sender-only: start every relayed log near its own last key	The sender cannot know what an offline subscriber holds; it is used only where it does know (the copy anchor)
Force base copies for ambiguous per-origin resumes	Reuses existing copy boundaries but replaces incremental catch-up with repeated full table copies; this task explicitly preserves the cursor work
Exact append-order resume for incremental per-origin starts	Needs RocksTransactionLogStore to mix exact and non-exact starts in one range, plus a per-log fallback; you chose the 60 s overlap (Q2 a)
Seed every unlisted origin from its direct seq row	A current member that is unlisted and unexcluded has partial coverage, and its direct cursor does not cover the tables it relays, so origins whose registry row still exists are not seeded

❓ Your call: Seeding a removed origin from its own connection's row assumes that direct subscription covered every table the relay now supplies. That coverage is the advertised one computeExclusionOrigins relied on, not a recorded one. If R's effective send scope was narrower and no relayed R entry reached this node since the upgrade, the seed suppresses relayed rows older than the cursor minus 60 s. I kept your seeding decision; the alternative is no seed, which replays every removed origin from 0 once per connection.

❓ Your call: The removal-only review found that probeNodeRow treats a physically absent registry row as deleted. A named config-route peer can be connected before its registry row arrives, so the unchanged membership predicate may seed a live peer with partial table coverage and suppress older relayed rows. Distinguishing a tombstone from an absent row is a separate cursor change; this scope update preserves the owner-specified cursor functions. Verify this membership case before promoting the PR.

❓ Your call: A per-origin cursor is not bound to the table scope it was recorded under. If a subscription later widens (a table turns on replicate, or a receivesFrom exclusion is removed) and the node reconnects without a base copy, a relayed origin's rows in the newly included table keyed more than 60 s below its cursor are not resent. Before this change those logs restarted at 0, so retained rows still arrived. The direct peer's own rows are already missed this way on main, because skipped records advance its scalar seqId, and nothing resyncs a widened scope. I left both cursors scope-unbound. Binding them to a scope fingerprint would fix both, and is a separate change. DESIGN item 26 records it.

⚠️ Look hardest: ORIGIN_CURSOR_OVERLAP_MS is not a proven bound. The incremental floor remains active for the life of the subscription. An entry appended more than 60 s after its key can therefore be skipped, including while the socket stays connected: this includes a delayed origin commit and an origin entry arriving at a relay over a lagging path. storage.maxTransactionOpenTime defaults to 30 s but can be extended, and commit-time write stalls are unbounded. A listed source's own resume already has this exposure with no overlap at all. #922's closed floor replaces both.

❓ Your call: Each committed frame with progress hands core one small cursor array (takeDurableOriginCursors). Publishing only at sequence updates would avoid it, but a sender sends those only after skipped records, so cursors would go stale. The core apply bench attaches a fresh cursor array per frame from onCommit and measured 147 µs per frame against 146 µs without (see Verification). There is no separate two-node catch-up throughput measurement.

❓ Your call: The closing review also identified same-socket resubscribe risks in the retained sender: resetting liveStartByLog while reusing auditLogIterable can leave a re-admitted origin bounded only by the predicate, walking its full retained log; a superseded loop can resume from back-pressure and overwrite the current loop's waker before parking. The adjudicator reproduced the range-map and waker coordination in deterministic in-memory checks against pinned core; there is no targeted cluster reproduction. This removal preserves the cursor and includeNodes implementation as instructed; assess these cases before promotion.

Product and architecture tour

Per-origin cursor lifecycle

How does a subscriber learn, keep and use its position in every origin's log?

Record, send, apply

The receiver records each origin's highest applied key per peer after commit. It sends those keys on the next subscription request, and the sender bounds each unlisted origin's log by them.

sequenceDiagram
    participant S as Sender (peer P)
    participant R as Receiver
    participant C as core seq row [seq, P]
    S->>R: frame (key K, origin O)
    R->>C: end_txn originCursors [[O, K]] after onCommit
    C->>C: nodes[O].originLogKey = max(old, K)
    Note over R: restart or reconnect
    R->>S: SUBSCRIPTION_REQUEST[4] { O: K }
    S->>S: startByLog O = K - 60 s, predicate floor
    S-->>R: only O entries keyed above the floor

Loading

Before	After
An origin the subscription neither lists nor excludes starts at 0 on every resubscribe.	It starts ORIGIN_CURSOR_OVERLAP_MS (60 s) below the subscriber's cursor for it; an origin with no cursor still starts at 0.

A cursor is per peer: it covers that subscription's table scope, so it is never borrowed from another peer's row.
An origin with a deleted or physically absent hdb_nodes row is seeded from its own connection's row. This assumes that connection covered the relayed tables; see the open membership concern above.

Base-copy tails resume in append order

Where can the sender prove what the subscriber holds, and so avoid the 60 s assumption?

• Fresh base copy. The copy holds every relayed entry committed before it starts, so each relayed log resumes past its own last committed key with resumeAfterExactStart. The receiver keeps those anchors (COPY_START[3]) as its cursors once the copy finishes. A relayed log whose boundary does not form loses only its own anchor and starts from 0.
• Copy-anchored logs drop their key filter, because a later append can carry an older key.

Re-inclusion after remove_node

How do a removed node's last writes reach a survivor that lacks them?

From removal to delivery

1. Removal — the survivor's hdb_nodes row for R is deleted, and the main thread drops R from its exclusion origins.
2. Include update — the survivor sends { includeNodes: [R], originCursors: { R: cursor } }.
3. Floor — the sender puts R's floor into the live range's startByLog and calls addLog.
4. Admission — core admits R's log at the next transaction boundary, and the sender loop is woken.
5. Delivery — R's rows keyed above the floor stream to the survivor without a resubscribe.

Files.

• replication/protocolCapabilities.ts: the originCursors level, its resolution and advertisement, and samePeerCapabilities.
• replication/replicationConnection.ts: the exported helpers (vector, floors, predicate, copy anchors, boundary probe), the receiver's recording and sending, and the sender's floors, copy-tail anchors and include-path wake.
• replication/DESIGN.md: Non-obvious behavior 26 and the capability table row.
• core: points at harper main (f7b766981), where the companion core PR is merged. That commit also brings in harper#3029, harper#3035 and harper#3030, which landed on harper main first; the remaining removed-origin cluster test and the harper-pro unit suite pass on it.
• integrationTests/cluster/fixture-origin-log/resources.js: LogEntryCount and OriginCursors read a node's per-origin log size and persisted cursors.

✅ Verification

End-to-end route: npm run test:integration -- --concurrency=1 integrationTests/cluster/removedOriginResumeCursor.test.mjs passes all 3 cases on rebuilt dist. An uncommitted two-node smoke also confirms that reconnecting after auditRetention logs a bounded base-copy resync and that a subsequent write converges. The smoke uses positive evidence (forced-copy warning and subsequent convergence); it does not exercise a retention-forced copy with relayed origins, so that combination remains a coverage gap. The retention-admission block is byte-identical to fetched origin/main; 12 cursor/floor/anchor functions, copy-tail code, incremental range, capabilities and the core pointer are unchanged by this scope update. The original fails-on-base evidence for per-origin resume is retained below.

• integrationTests/cluster/removedOriginResumeCursor.test.mjs. A, B and R form a full mesh, and Q replicates only with A, so Q reaches B only by relay.
  • OLD rows are written, then MID rows more than 60 s later. B stops and R writes LATE rows that reach A only. R is removed on A, and then on B.
  • B gets LATE through A as soon as it removes R, without a restart. No peer sends OLD rows (R's, or Q's through A) when B or A restarts, and B's persisted cursor for Q on its A row covers MID.
  • A node N added afterwards by base copy from A and then B gets no OLD row in either copy's tail. Its restart pulls no OLD row either. R's log on A and B does not grow; N's log may append the overlap-window rows once, bounded by the MID and LATE row count.
  • The oracle is the sender's own wrote record <id> … remoteNode <peer> debug line. The restart checks assert absence without a matching positive send control, and missing log files read as empty; strengthening that oracle remains open. The existing fails-on-base evidence below establishes that it observed the defect in the original runs.
  • Base (a3e1a9d0 + core 5230512f9): 3/3 tests fail. B is resent R's OLD rows on restart, and N's copy tail carries all 20. Fixed build: green on the final head, and 3/3 runs earlier (5/5 before the include-path assertion was added).
• Unit tests.
  • unitTests/replication/originCursors.test.mjs (new): vector construction, including per-peer rows, removed-origin seeding through the helper's membership predicate, and malformed rows. Also covers floor resolution, which rejects unknown or inherited names and never mints an id, the subscription predicate, copy anchors, the per-log boundary fallback.
  • unitTests/replication/protocolCapabilities.test.mjs: originCursors resolution, the fallback for an absent or malformed bag, and advertisement only with per-origin logs.
  • unitTests/replication/replicateFalseSendPaths.test.mjs: passes the new capability argument explicitly.
• Full suites. Scope-update verification: harper-pro npm run test:unit: 1668 passing, 2 pending. npm run build, npm run typecheck, and changed-file Prettier check pass. npm run lint reports the same 24 warning signatures as fetched origin/main (no new warnings). Core companion verification before this scope removal: test:unit:resources 4019 passing, 54 pending. Full integration gate on Node 26.2: 217 passing, 7 failing, 3 cancelled, 15 skipped. Fetched main (f7aacb6b, core 5230512f9) reproduces the worker-placement, non-replicated-table copy, and static-redeploy failures. The other three failed files (replicationWedgeRecovery, selectiveTableSubscription, subscriptionSetupRecovery) pass on main and in a focused branch rerun (6/6 cases); the full gate is recorded as failed. Removed-origin integration also passed all 3 cases again within the full run.
• Apply-path cost (core companion; bench-run, CPU capped at 2 GHz, 20k frames, 3 rounds, median µs per frame):
  • Advancing frames: 146 without cursors, 147 with cursors.
  • Repeated-localTime frames: 99 with no cursors (the same work as main), 99 with a cursor that did not rise, 111 with a rising cursor (one cursor write).

Independent review. Claude, Gemini and Cursor Composer completed full finding passes, followed by harper-domain adjudication. The removal is faithful. The domain reviewer rates the retained session-long 60 s floor as a blocker and membership/sender-lifecycle risks as major; those paths remain unchanged under the owner instruction, and their specific promotion concerns are above. The Pro allocation cost remains unmeasured despite the existing core apply benchmark. Parallel startup failure can also strand test nodes before the suite registers them. The required planning recheck cleared the removal-only framing; review does not imply the draft is ready to merge.

CI on cb31e647. Builds, unit tests (Node 22/24/26), typecheck, lint, YCSB, all non-cluster integration shards, copy-gap regressions, and five of six cluster shards pass. Cluster shard 1/6 fails because both replicateFalseFullCopy suites cannot observe a restart within 60 s; main has the same failure in shard 6/6, at f7aacb6b. Deleting the idle test shifted that file's shard assignment. No new review comments arrived during the watch.

No documentation companion is needed: this scope update changes internal resume admission and adds no public API or configuration.

Refs #989

Depends-on: HarperFast/harper#3080

_{🤖 Generated by Claude Code (original implementation) and OpenAI Codex (scope update); posted via @kriszyp.}

Related PRs: #554 overlaps (copy-tail boundaries; reconcile its checks with the relayed-anchor fallback), #717 overlaps (held frames must keep skipping onCommit, where origin cursors are noted), #815 overlaps (capability registry; admission now waits for a transaction boundary), #959 overlaps (sender loop wait and its include waker), #983 overlaps (sender loop waker; moves the core pointer), #940 overlaps (capability registry), #956 overlaps (capability registry; moves the core pointer), #995 overlaps (moves the core pointer), #981 overlaps (moves the core pointer), #992 overlaps (moves the core pointer), #997 overlaps (moves the core pointer), #988 independent, #991 independent, #996 independent, #990 independent, #943 independent (this branch is rebased on it), #987 independent, #984 independent, #986 independent, #979 independent, #985 independent, #942 independent, #955 independent, #800 independent
Complexity: complicated

Origin and current scope

The original dispatch covered #989. The current owner-directed scope addresses removed and relayed origins, preserving per-origin recording/sending, the 60 s overlap floors, includeNodes and exact relayed-log anchors for base-copy tails. It deletes no transaction-log entry, log store or sequence row. The core companion #3080 is merged and the core pointer is unchanged by this scope update.

_{Dispatch: task harper-pro-989 · queued by kris-session · ran by claude/opus/xhigh · worker kzyp-xps-1}

_{Review-Coverage: authored=codex; ran=cursor-composer,gemini,claude; adjudicated=domain; declined=cursor-grok,cursor-kimi,cursor-muse; rounds=11; full=5 @ cb31e64}

_{Review-Attention: deep ~80m (critical: protocolCapabilities.ts, replicationConnection.ts; decisions: do-less-alternative, removed-origin-coverage, scope-widening-policy; raised: earlier round at this SHA) @ cb31e64}

[gemini-code-assist]

Code Review

This pull request implements per-origin resume cursors (harper-pro#989) to allow removed and relayed origins to resume incrementally instead of starting from zero. It introduces the originCursors capability, records origin progress on the receiver, transmits cursor vectors during subscription requests, and applies a 60-second overlap floor on the sender. Additionally, it updates retention checks to support incremental resumes for idle peers and adds comprehensive integration and unit tests. The review feedback highlights a potential crash in collectSeqRows where entry.key is accessed outside of the try-catch block, suggesting wrapping both key and value extraction to ensure robustness against decoding errors.

[github-actions]

Release cherry-pick v5.3: failed

The cherry-pick job for v5.3 did not complete. It is unclear whether v5.3 was updated — check before re-running.

Run: https://github.com/HarperFast/harper-pro/actions/runs/37662733435

Re-run once the cause is addressed (workflow_dispatch, pr_number=998).