Anchor the replication base copy on a log-order barrier instead of a timestamp

[kriszyp]

The base copy published a resume cursor that was not a boundary in the log it resumes from, so a transaction in flight when a node joined could be delivered by neither the copy nor the audit tail. It was not delayed: it was never replicated to that peer, and nothing later repairs it. Full trace in #876.

Why a timestamp could never work here

Two orderings disagree, and the copy assumed they agreed:

	fixed when
a transaction's log key	the transaction is created
its log batch	the transaction commits

So the log is written out of key order, measured at 47 inversions per 1000 concurrent transactions across 4 threads. Take a transaction created at T-0.2 and committed at T+0.02, against an anchor of T. It sits behind the anchor numerically and after it physically. The walk had already passed its row, and the tail's per-entry predicate excluded it: timestamp > start, in rocksdb-js's TransactionLog.query, where _findPosition only seeks.

The fix

Anchor on the latest committed entry of the local log, and resume past it in append order. Just before COPY_START, the leader finds the latest committed key with findLastCommittedLogKey and sends it as copyStartTime. Nothing is written to form the boundary, so this no longer needs a core change. The earlier copyBarrier entry and its companion harper PR, harper#2697, are dropped, and the core gitlink is back on main's.

Why the last committed entry is a boundary. rocksdb-js yields entries only up to its committed-read watermark. That watermark advances past a transaction only after its RocksDB Commit() returns ok (transaction.cpp, the commitFinished call gated on status.ok()), and only across a contiguous prefix. So everything up to the anchor is visible to the walk, and every transaction still in flight appends after it, whatever its key. A real-log test pins that property.

Finding the last entry without reading history. The log reads only forward, but a range seeks by key through each file's running-max index. The first entry it yields sits at the seek position, so the helper bisects on key, pulling one entry per probe: about 30 probes, and no scan. An earlier revision scanned from the last-flushed position, which found nothing after a memtable flush and fell back to the wall clock on most RocksDB copies. A widening-window variant could scan a quiet log's whole history. The cluster runs caught the first problem and review caught the second.

The tail resumes with exactStart + resumeAfterExactStart (boundary range). Once the key matches, the predicate stops consulting the key at all. The range is built before the walk and proven with a throwaway pull under the same options. exactStart is scoped to local by startByLog, so peer logs in the aggregate are untouched.

A resumed copy tries the anchor as an exact entry first (classifyResumeAnchor): an exact match resumes in append order, and anything else keeps the timestamp resume. That anything else covers a pre-#876 Date.now() cursor, a purged entry and an empty-log copy. COPY_ORDER_VERSION stays 1, so a rolling upgrade honours in-flight copies instead of restarting them.

An empty log tails from its start with an ordinary range: everything it will yield commits later.

Gated on the store, not the config. The gate is auditStore.reusableIterable === true && logName === 'local', not STORAGE_IS_ROCKSDB. That flag is a config snapshot that misreports the engine in a worker thread, and it previously let an LMDB table reach a RocksDB-only call.

Kept from earlier rounds. The anchor is validated on both peers (follower). The sender's cursor and SENDING_TIME_POSITION only climb, since append order is not key order. A tail-time boundary failure rebuilds an ordinary range immediately. A degraded copy says so at warn. Design note: replication/DESIGN.md.

For the human reviewer

1. This PR's approach changed in this revision, on your call in the dispatch thread. It now reads the last committed entry instead of writing a barrier. I skipped a fresh --mode plan framing gate, because the owner chose the framing directly. The framing verdicts recorded below were against the barrier design and are kept as history.
2. Empty-log copies still send Date.now() to the follower (why). The leader's in-connection tail is correct: it reads from the log's start. A mid-copy disconnect on an empty log, though, resumes by timestamp and can miss a transaction that was in flight. A low sentinel such as 1 would close that, but shouldForceBaseCopyForRetention reads it as purged history. It would then recopy an idle follower on every reconnect after COPY_COMPLETE. Closing it properly needs a mode flag in the cursor, the same durable-cursor work as Replication resume cursors need an append-order mode to survive a reconnect or a relayed origin #879.
3. Still not covered (unchanged scope, Replication resume cursors need an append-order mode to survive a reconnect or a relayed origin #879): a reconnect after COPY_COMPLETE resumes by timestamp, because matchesSubscription requires startTime below an entry's key. Relayed-origin logs are not anchored either.
4. Degrade-open, not fail-closed. An unreadable log, an unformable boundary, or an anchor naming no entry all fall back to the pre-Replication base copy can silently drop a transaction that is in flight when a node joins #876 timestamp range and log it. Fail-closed was tried in an earlier round and withdrawn, because it produced reconnect loops that copied nothing.
5. Declined review findings, with evidence:
   • "Bisection probes leak iterators": refuted. An early return inside for…of calls the iterator's return() (the language's IteratorClose step).
   • "Fractional probe keys": refuted. Log keys are fractional doubles, and findPositionByTimestamp takes a double.
   • "A poisoned persisted cursor loops 1008 against an old leader": kept at minor. No shipped leader emits an invalid anchor, so reaching this needs already-corrupt storage.
   • "LMDB and relayed copies warn on every copy": kept deliberately. The warning is what exposed an inert gate before.
6. Also fixed on the way: an earlier rebase of this branch had dropped main's Flaky: clone from a legacy v4 leader intermittently fails when audit-log forwarding throws RangeError converting a non-integer to BigInt #737 paragraph from replication/DESIGN.md, so it is restored. The unit-test exit-flush override is reverted, because main fixed its cause in Remove the unit-test root only after its databases are closed #893 and the override masked that change's lifecycle test. The txnlogTearReplication oracle loosening is reverted, because it existed only to tolerate barrier frames.

Earlier rounds (barrier design), kept for the record

• The planning gate returned better-alternative-exists twice. Round 1's alternative, reconstructing an in-flight floor from registryStatus(), was refuted by rocksdb-js assigning startTimestamp before it publishes the handle, and that design was replaced. Round 2's corrections were adopted: the pre-positioned iterator, validating the anchor on both peers, and keeping a resumed copy's original anchor.
• The first implementation was inert in production. It was gated on !excludedNodes, and excluded is always an array, so the gate was never true. Every test passed against the inert path. That is why verification asserts on the absence of the fallback warning, not on a green test.
• matchesSubscription does not filter the copy's tail. It has one call site, gated on the per-origin subscribedNodeIds[id].timeRange, and auditSubscription.startTime is written, never read, on that path.

Verification

• Defect and fix, against a real RocksDB log (copyAnchorResume.test.mjs). A held-open transaction takes the lower key but appends last. The timestamp range drops it, while resuming past the last committed entry delivers it. The log is flushed first, so the anchor cannot depend on post-flush entries; this pins the regression the first revision had. A third test polls the log while 200 transactions commit, and asserts every yielded entry is already visible to a read.

• npm run build and tsc --noEmit are clean. prettier is clean on touched files. oxlint shows only the pre-existing subscriptionRequest warning, on an untouched line.

• npm run test:unit: 1258 passing, 0 failing, 1 pending.

• Cluster integration, one file at a time on a private TMPDIR, with instance logs kept (HARPER_INTEGRATION_TEST_LOG_DIR):

  file	result	RocksDB base copies	timestamp fallbacks
  addNodeFullCopy	pass 1 / fail 0	2	0
  copyResumeCursorGap	pass 1 / fail 0	1	0
  directionalFlowReplication	pass 1 / fail 0	1	0
  txnlogTearReplication (oracle now back to main's)	pass 1 / fail 0	4	0
  fullyConnectedReplication (RocksDB + LMDB suites)	pass 10 / fail 0	36	0 (LMDB: 12, expected)

  Warn level is logged in these runs, so zero fallback lines means each RocksDB copy took the log-order path. An earlier run of the same table is what exposed the flushed-log gap: 25 fallbacks on fullyConnectedReplication alone.

• Independent pre-push review, 4 rounds on this revision: 1 full round (codex + gemini + cursor-muse, domain adjudicator timed out) and 3 deltas. The final delta ran codex + gemini + cursor-muse + harper-domain and found nothing new. Everything left is an accepted-known limit or a decision listed above.

Fixes #876

Origin — the dispatch brief this PR was written from

Anchor the replication base copy on a log-order barrier instead of a timestamp

LIVE CONVERSATION about #878.

You are answering a person, in a thread, one turn at a time. Every turn:

1. Read the whole thread in this dispatch file's # Log — it is the conversation so far, and
   each of your previous turns is in it. Read the PR/issue and the code as needed.
2. Answer the LAST message. Append your answer to # Log as your turn. Prose, not a report:
   they are talking to you, and a status template is not an answer.
3. Set status: needs-input and stop. The thread stays open; their next message resumes it.

Each turn arrives as ASK (answer it, change nothing) or PERFORM (do it, then say what you did) —
the person chose which when they sent it, and the run's own prompt tells you which one this is.
Never infer it from the wording: an unrequested commit in the middle of a discussion and a polite
description of work that was supposed to happen are the two failures this exists to prevent.

Never mark a PR ready and never merge from this conversation.

_{Dispatch: task chat-pr-harper-pro-878-kriszyp · queued by unknown · ran by claude/opus/high · worker kzyp-xps-1}

_{Review-Coverage: authored=claude; ran=cursor-muse,codex,gemini; adjudicated=domain; declined=cursor-grok,cursor-composer,cursor-kimi; rounds=31; full=4 @ 716cea8}

_{Human-Review-Need: 4 (decisions: empty-log-wall-clock-cursor, degrade-open-on-boundary-failure, reconnect-gap-deferred, no-cursor-version-bump, anchor-local-log-only) @ 716cea8}

[gemini-code-assist]

Code Review

This pull request introduces a more robust mechanism for base-copy resume boundaries in replication by replacing timestamp-based anchors with log-order barriers. This change ensures that transactions committed during a bulk copy are correctly captured, preventing data loss. The implementation includes new utility functions for barrier management and anchor classification, along with comprehensive unit tests. I have reviewed the provided feedback and kept the suggestion regarding the withDatabase helper, as it correctly identifies a potential resource leak in the test cleanup logic.

[gemini-code-assist]

In the withDatabase helper, if RocksDatabase.open throws an error, the finally block is not entered, and the temporary directory created by mkdtempSync is leaked on disk. Wrapping the database initialization and log usage in nested try-finally blocks ensures that the temporary directory is always cleaned up properly.

Suggested change

	async function withDatabase(run) {
	const dir = mkdtempSync(join(process.env.TMPDIR || tmpdir(), 'copy-barrier-'));
	const db = RocksDatabase.open(join(dir, 'db'));
	try {
	await run(db, db.useLog('local'));
	} finally {
	db.close();
	rmSync(dir, { recursive: true, force: true });
	}
	}
	async function withDatabase(run) {
	const dir = mkdtempSync(join(process.env.TMPDIR || tmpdir(), 'copy-barrier-'));
	try {
	const db = RocksDatabase.open(join(dir, 'db'));
	try {
	await run(db, db.useLog('local'));
	} finally {
	db.close();
	}
	} finally {
	rmSync(dir, { recursive: true, force: true });
	}
	}

References

1. In test cleanup hooks (such as after or afterEach), guard cleanup functions against uninitialized or undefined variables to prevent synchronous errors from blocking subsequent cleanup steps and causing resource leaks.

[claude]

Suggestion (non-blocking): RocksDatabase.open(...) runs outside the try, so if it throws, the temp dir created by mkdtempSync on the prior line is never cleaned up by the finally below. This is the same leak gemini flagged on this helper under its old name (copyBarrierResume.test.mjs, now-outdated thread) — it persisted through the rewrite into this new file. Nest the open call in its own try/finally:

Suggested change

	async function withDatabase(run) {
	const dir = mkdtempSync(join(process.env.TMPDIR || tmpdir(), 'copy-anchor-'));
	const db = RocksDatabase.open(join(dir, 'db'));
	try {
	await run(db, db.useLog('local'));
	} finally {
	db.close();
	rmSync(dir, { recursive: true, force: true });
	}
	async function withDatabase(run) {
	const dir = mkdtempSync(join(process.env.TMPDIR || tmpdir(), 'copy-anchor-'));
	try {
	const db = RocksDatabase.open(join(dir, 'db'));
	try {
	await run(db, db.useLog('local'));
	} finally {
	db.close();
	}
	} finally {
	rmSync(dir, { recursive: true, force: true });
	}
	}

[claude]

Reviewed; no blockers found.