Add cluster integration test: sourcedFrom replicationSource: true (§5.5 Cat 15)

[kriszyp]

Summary

• Adds integrationTests/cluster/cacheReplicationSource.test.mjs — the last uncovered sub-case of the v5 Integration Test Plan §5.5 Caching / Category 15 (tracked in harper#1189 which noted it requires a 2-node cluster and could not be added to the single-node caching.test.ts).
• Adds companion fixture fixture-cache-repl-source/ with an OriginCache table declared sourcedFrom(OriginSource, { replicationSource: true }).

What the test asserts

A lightweight mock HTTP "origin" server runs in the test process and counts per-node fetch calls. Two Harper nodes start with the fixture pre-loaded:

Assertion	Description
(a)	Cache miss from nodeB contacts origin exactly once (stampede prevention applies cluster-wide). A TODO marks where the stricter per-node check (fetch routed only to nodeA) should land once replicationSource routing is implemented in core.
(b)	The cache entry is byte-identical on both nodes after replication converges.
(c)	A second request on nodeB is served from the replicated cache — zero additional origin fetches.
(d)	A separate id fetched from nodeA also replicates to nodeB and is served from cache on nodeB without a re-fetch.

Mapping to v5 Integration Test Plan

§5.5 Caching / Category 15 — replicationSource: true sub-case. The single-node core/integrationTests/server/caching.test.ts left a TODO ("requires a 2-node cluster setup") that this PR closes.

Passing output

▶ Cache replicationSource cross-node
  ✔ (a) origin is contacted exactly once on cache miss from nodeB (31ms)
  ✔ (b) record is byte-identical on both nodes after replication converges (268ms)
  ✔ (c) second request on nodeB is served from replicated cache without re-fetching origin (8ms)
  ✔ (d) separate id also caches and replicates correctly (291ms)
✔ Cache replicationSource cross-node (9188ms)

Notes

• The startHarper (vs setupHarperWithFixture) approach was necessary to ensure the pre-allocated loopback address matches the replication.securePort in the config. setupHarperWithFixture overwrites ctx.harper before calling startHarper, causing the hostname to be re-allocated and resulting in a securePort mismatch (silent ECONNREFUSED).
• The replicationSource: true option is stored in sourceOptions but the routing logic (origin fetch only on the designated source node) is not yet implemented in core. The TODO in test (a) tracks the stricter per-node assertion.
• No core submodule changes.

Generated by Claude Sonnet 4.6 (claude-sonnet-4-6) as a subagent.

[gemini-code-assist]

Code Review

This pull request introduces a new cross-node integration test and associated fixture files to verify the behavior of the sourcedFrom cache with replicationSource: true in a two-node cluster. The review feedback highlights several critical improvements: resolving loopback addresses sequentially instead of concurrently to prevent a race condition, wrapping rawOperation calls in try/catch blocks to ensure retry loops do not crash on network errors, and using fileURLToPath to safely resolve paths in ES modules without relying on CommonJS globals.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

Add fileURLToPath import from node:url to safely resolve the installation script path in ES modules without relying on CommonJS globals.

Suggested change

	import { startHarper, teardownHarper, getNextAvailableLoopbackAddress } from '@harperfast/integration-testing';
	import { resolve, basename, join } from 'node:path';
	import { sendOperation, fetchWithRetry } from './clusterShared.mjs';
	import { startHarper, teardownHarper, getNextAvailableLoopbackAddress } from '@harperfast/integration-testing';
	import { resolve, basename, join } from 'node:path';
	import { fileURLToPath } from 'node:url';
	import { sendOperation, fetchWithRetry } from './clusterShared.mjs';

[gemini-code-assist]

In ES modules (.mjs), the CommonJS module global is not defined. If this test is run on a Node.js version that does not support import.meta.dirname (Node.js < 20.11.0), the fallback to module.path will throw a ReferenceError. Using fileURLToPath(import.meta.url) is the standard, cross-version compatible way to resolve paths in ES modules.

Suggested change

	process.env.HARPER_INTEGRATION_TEST_INSTALL_SCRIPT = resolve(
	import.meta.dirname ?? module.path,
	'..',
	'..',
	'dist',
	'bin',
	'harper.js'
	);
	process.env.HARPER_INTEGRATION_TEST_INSTALL_SCRIPT = resolve(
	fileURLToPath(import.meta.url),
	'..',
	'..',
	'..',
	'dist',
	'bin',
	'harper.js'
	);

[gemini-code-assist]

If rawOperation throws a network error (such as ECONNREFUSED because the server is still starting up), the error is uncaught and the retry loop will terminate immediately on the first attempt. Wrapping the call in a try/catch block ensures the retry logic functions as intended.

	for (let i = 0; i < 20; i++) {
		try {
			const result = await rawOperation(nodeA, {
				operation: 'create_authentication_tokens',
				authorization: nodeA.admin,
			});
			if (result.status === 200 && result.body.operation_token) {
				tokenResp = result.body;
				break;
			}
		} catch (err) {
			if (i === 19) throw err;
		}
		await delay(300);
	}

[gemini-code-assist]

Similar to the first loop, if rawOperation throws a network error, the loop will crash immediately instead of retrying. Wrapping this in a try/catch block ensures robust retry behavior.

	for (let i = 0; i < 30; i++) {
		try {
			const result = await rawOperation(nodeB, {
				operation: 'add_node',
				rejectUnauthorized: false,
				hostname: nodeA.hostname,
				authorization: 'Bearer ' + tokenResp.operation_token,
			});
			// Success or "already exists" are both fine; ECONNREFUSED means replication
			// server not ready yet — wait and retry.
			if (result.status === 200) break;
			const errMsg = JSON.stringify(result.body);
			if (errMsg.includes('ECONNREFUSED') || errMsg.includes('ECONNRESET') || errMsg.includes('connect ')) {
				await delay(500);
				continue;
			}
			// Other 5xx: unexpected, propagate
			throw new Error(`add_node failed (${result.status}): ${errMsg}`);
		} catch (err) {
			if (i === 29) throw err;
			await delay(500);
		}
	}

[claude]

The prior blocker (orphaned node handle on partial startup failure) has been correctly resolved by commit b4e3ced. The .then(() => { ctx.nodeA = ctxA.harper; }) pattern captures each handle immediately when its startHarper call resolves, independent of the sibling — so the after() guard (ctx.nodeA && teardownHarper(...)) cleans up whichever nodes successfully started.

One minor robustness nit remains but is not a blocker:

1. origin.close() is unguarded against a failed startMockOrigin()

File: integrationTests/cluster/cacheReplicationSource.test.mjs:291

What: In after(), origin.close() is called unconditionally. If startMockOrigin() threw (the very first await in before()), origin is undefined and after() will throw a TypeError, shadowing the original error.

Why it matters: In practice startMockOrigin() is extremely unlikely to fail (it binds to port 0 on loopback), so this is a very low-probability scenario. It is not a correctness bug in the happy path. Worth a one-line guard but not a merge blocker.

Suggested fix:

await origin?.close();

No other blockers found. The rest of the PR — fixture files, schema, resource, cluster connectivity logic, and per-test assertions — is solid.

[kriszyp]

Addressed the orphaned-node concern from the review thread: node handles are now captured inside .then() callbacks chained directly on each startHarper call, so each handle is assigned as soon as that node resolves — independent of its sibling. If one start fails, the successfully-started node's handle is still captured and the after() teardown guard (ctx.nodeA && teardownHarper(...)) will clean it up correctly. Commit: b4e3ced.

— Claude Sonnet 4.6