Add a deterministic cluster test for the relayed-origin resume gap

[kriszyp]

Adds integrationTests/cluster/relayedOriginResumeGap.test.mjs, a deterministic 4-node cluster test for the relayed-origin resume gap (the W2 acceptance test proposed on #432). Until now only the intermittent 4-node backlogRecovery stress test ever reproduced this class: a node that holds an origin's rows delivered only through a relay has no resume cursor for that origin at all, so when its later subscription to that origin resolves cursorless the pre-#428 now - 60s start silently dropped the backlog written while the relay was down. The test forces that condition on every run — a chain O ← R ← T plus a witness W on O with databases: ['data'] so the system database cannot re-mesh the chain, asserted up front: T holds every O-origin row yet has no connection to O; R killed with no grace, the backlog written and aged past the legacy floor, then T adds O directly as a non-leader — and asserts exact-count convergence on all four nodes plus the resume decision in T's log (start time: 1 for O, then the #428 full-copy request). One env-gated hook, maybeLegacyCursorlessResumeStartForTest (HARPER_TEST_DISABLE_CURSORLESS_FULL_COPY=1, default-off, unit-tested), supplies the pre-#428 start as an override only; look hardest at the else branch, which keeps the production startTime = 0 and its two warn lines unchanged. The same file doubles as a red-proof: with the hook injected into T and pinned off on every other node (every child inherits the runner's environment, so each control node's log is asserted free of the legacy resume), the scenario asserts the loss signature — a live T←O link, no phase-2 row on T and T frozen below O and W, and still frozen below every peer after R returns.

For the human reviewer

1. Red-proof mode ships with no CI consumer. The env hook and the HARPER_TEST_DISABLE_CURSORLESS_FULL_COPY=1 mode only run when someone runs them by hand; CI's cluster job runs the green mode. Kept because the dispatch brief asked for a hook-gated, CI-excluded red-proof so anyone can re-run the loss demonstration, and because W2's gap detection is meant to flip that mode green as its own acceptance signal. Alternative: drop the hook and rely on the green test's two log discriminators. Reversal is mechanical (delete the helper, its unit test, and the PROTECTION_DISABLED branches).
2. The 65 s ageing wait runs in the green mode too. Gating it on the red-proof would save 65 s per CI run, but then the green test's convergence assertion would also pass under the hook (phase 2 would be younger than the legacy floor), so "the same assertions go red when the protection is disabled" would hold only for the log oracle, not for the count. The wait is the scenario's clock, not a poll. One-line change either way.
3. R is killed with zero grace rather than stopped cleanly. killHarper({ graceMs: 0 }) sends SIGTERM followed at once by SIGKILL, the closest the harness gets to the OOM kill the stress scenario models. The trade: after an abrupt kill R may come back with its persisted O cursor or resume cursorless and full-copy, so test 2 proves R converges without pinning which mechanism did it (its wait is named accordingly). A clean stop would make that deterministic at the cost of crash realism. Trivially reversible.
4. The resume decision is asserted from log text. The oracle pins start time: 1 (a debug line, hence logging.level: 'debug' on all four nodes), the full-copy warn line, and the hook's warn line. There is no non-log way to observe the decision today; adding one would be a small product API change outside this test-only brief. W4's per-origin cursors are expected to retire the start time: 1 discriminator anyway.
5. The fourth node is a witness on O, not a peer downstream of T (author's call, not from the review). Base-copied rows are audit-less snapshots on the receiver, so a peer subscribed to T's audit log would never receive the rows T recovers by full copy and could not converge. W instead proves O's backlog is live and replicable during the whole run. Worth knowing when W2 designs its heal path: a relay cannot heal a node that is directly connected to the origin, which the red-proof's second test pins.

Verification

• End-to-end route: the new integration test itself. npm run test:integration -- integrationTests/cluster/relayedOriginResumeGap.test.mjs passed 7/7 runs on this branch (five on the first commit: 81 s, 119 s, 105 s, 82 s, 82 s; one after each review-fix commit: 82 s, 86 s; the slower runs overlapped other suites). The first test carries the 65 s ageing window, the second restarts R and converges all four nodes.
• Red-proof, hook-gated mode: HARPER_TEST_DISABLE_CURSORLESS_FULL_COPY=1 npm run test:integration -- integrationTests/cluster/relayedOriginResumeGap.test.mjs passed 4/4 runs, asserting T frozen at 21 rows (phase 1 + the live write) while O and W hold 51, start time: 1 for O in T's log with no full-copy request, no legacy-resume line in any control node's log, and T still at 21 after R returns.
• Scratch run of the green assertions with the hook forced on T (a two-line local edit, not committed): both tests go red with the frozen-count signature — Timed out after 90000ms waiting for T convergence through its direct O subscription: every node at 50, last saw {"O":50,"T":20,"W":50} and, after R returns, last saw {"O":50,"R":50,"T":20,"W":50}.
• Unit gate: npm run test:unit — 845 passing, including the 4 cases in unitTests/replication/maybeLegacyCursorlessResumeStartForTest.test.mjs.
• Neighbouring cluster tests on the edited branch: proxiedResumeBacklog, proxiedLeadingDuplicateSkip, addNodeFullCopy, reverseBaseCopyDeclined — all pass. npm run lint:required clean; prettier applied to the changed files. The full test:integration:cluster suite is left to CI.
• CI on this PR: the new test ran in cluster shard 6/6 and passed (66.6 s + 3.7 s). Cluster shard 1/6 is red on txnlogTearReplication ("B receives the entries written after a torn frame in A's local log"), the same failure main's latest integration run (33548173687) shows on all three Node versions, so it is pre-existing and unrelated to this change.

Refs #432

Complexity: medium

🤖 Generated with Claude Code

https://claude.ai/code/session_01BoGMRMSjqsUHVetXqfrzVL

_{Review-Coverage: authored=claude; ran=codex; blocked=gemini(auth); declined=cursor-grok,cursor-composer,domain; rounds=3 @ 8a13685}

_{Human-Review-Need: 4 @ 8a13685}

[github-actions]

Release cherry-pick v5.2: merged

Cherry-picked onto v5.2.

[gemini-code-assist]

Code Review

This pull request introduces a new integration test, relayedOriginResumeGap.test.mjs, and associated unit tests to verify the relayed-origin resume gap behavior, ensuring that a node without a resume cursor requests a full copy rather than silently resuming from a legacy timestamp. It also adds a test-only override function, maybeLegacyCursorlessResumeStartForTest, to simulate the legacy behavior when a specific environment variable is set. The review feedback suggests improving the parallel process startup in the integration test by assigning successfully started instances within their respective promise chains inside Promise.all to prevent resource leaks if one of the processes fails to start.

[gemini-code-assist]

When starting multiple asynchronous processes in parallel using Promise.all, assign the successfully started instances within their respective promise chains (e.g., using .then()) rather than waiting for Promise.all to resolve. This ensures that if one process fails to start, the already started processes are still recorded in ctx and can be properly cleaned up during teardown, preventing resource leaks.

			await Promise.all(
				contexts.map((nodeCtx, i) =>
					startHarper(nodeCtx, nodeConfig(hostnames[i], hookEnv(roles[i] === 'T'))).then(() => {
						ctx[roles[i]] = nodeCtx.harper;
					})
				)
			);

References

1. When starting multiple asynchronous processes in parallel using Promise.all, assign the successfully started instances within their respective promise chains (e.g. using .then()) rather than waiting for Promise.all to resolve. This ensures that if one process fails, the already started processes are still recorded and can be properly cleaned up during teardown.