Keep fault-injected cluster test nodes off the replication pool, fixing main's pool-leg failures

[kriszyp]

⊙ Problem

Main's Integration Tests have been red on every run since Run replication on a dedicated worker pool (replication.threads). That PR added a cluster matrix with the pool on (HARPER_CONFIG={"replication":{"threads":2}}) for every node. Two failure families repeat on every pool leg (Node 22, 24 and 26.5) and never on a non-pool leg. I checked six main runs, from 37883946911 to 37994119386:

• copyGapCursorBanking — fault schedule did not fully materialize (0/3 injected failures). The copy itself converged: 40/40 records, every blob payload intact.
• truthResiduals R1 and R5 — the killed worker must be the one that owns the subscription, and the same check for the hook-arming worker.

Both are test-harness failures, not product defects. Each suite injects its fault from application code: a blob-save injector in one, a worker-kill endpoint and a subscribe-deferral switch in the other. Pool workers load no application code (replication/DESIGN.md). With the pool on, the thread that owns the replication socket never runs the injector. The injector is armed on an HTTP worker that does not own the socket.

❓ Your call: is turning the pool off on the fault-injected node the right requirement, instead of reverting #983? Recommendation: yes. Nothing in the failures points at #983's product code; they come from the test layout #983 introduced.

💡 Solution

A new replicationPoolOffEnv() overrides only replication.threads to 0 in the inherited HARPER_CONFIG, and keeps every other key. It applies only to the node that carries the injector: node B in the copy-gap suite and the subscriber in truth-residuals. The peer keeps the pool, so the pool legs still run these suites against a mixed cluster.

⚖️ Alternatives

• Revert Run replication on a dedicated worker pool (replication.threads) #983: rejected. The behavior Run replication on a dedicated worker pool (replication.threads) #983 added is correct here; only these two tests assume HTTP-worker ownership.
• Skip R1/R5 and the copy-gap test when the pool is on: rejected. It drops the pool-on peer coverage that the mixed layout keeps.
• Make fault injection reach pool workers (for example, an env-armed preload that every thread type loads): this is the only way to test a blob-save fault, or an owner-worker death, inside a pool worker. It is a larger harness change, so I left it as a follow-up.

❓ Your call: no integration test covers a fault inside a pool worker, before or after this PR. The worker-exit truth stamp in replication/subscriptionManager.ts (ensureWorkerExitHandler) does not depend on worker type, so I expect it holds for a pool worker, but no test proves that. Should a follow-up add a pool-reachable injector?

✅ Verification

• Built dist at main 91655cc plus this change. Ran each suite once locally with HARPER_CONFIG={"replication":{"threads":2}}:
  • truthResiduals: pass 5, fail 0.
  • copyGapCursorBanking: pass 1, fail 0. Log: injected=3 bankedReconnects=2 resumeKeys=[13,29] B records=40/40 inPlaceRepairs=2.
• Not fixed here, because none fails on every run: removedOriginResumeCursor (1 failure in 6 runs), "take down the central node" (2 failures, both pool legs), and "replicate per-record expiration" (1 failure). The shard 6/6 origin-floor failures in run 37884785517 also hit non-pool legs and have not recurred since harper-pro#1017.
• Pre-push review (codex, gemini, cursor-composer, Harper domain): no correctness defects.
  • Gemini's "threads may be an object": dropped. CI sets a number.
  • "B may have several HTTP workers, so arming could hit the wrong one": moot. B's log shows the injector installed once, so B runs a single HTTP worker.
  • Comment-trim nit and the poolEnv (recordLockShared.mjs) merge suggestion: declined as low stakes.

Refs #983

— Claude Opus 5.5

🤖 Generated with Claude Code

https://claude.ai/code/session_019wVt5YqGDKbMBJNfGGJqGS

Related PRs: #1013 overlaps, 18 others independent

Origin — the dispatch brief this PR was written from

Investigate Harper-Pro replication-pool CI regressions after #983

_{Dispatch: task ci-regression-harper-pro-983-20261009 · queued by unknown · ran by claude/opus/medium · worker kzyp-xps-1}

_{Review-Coverage: authored=claude; ran=cursor-composer,gemini,codex; adjudicated=domain; declined=cursor-grok,cursor-kimi,cursor-muse; rounds=1; full=1 @ d5c4f06}

_{Review-Attention: skim ~2m (decisions: pool-off-on-faulted-node, helper-in-clustershared) @ d5c4f06}

[gemini-code-assist]

Code Review

This pull request introduces the replicationPoolOffEnv helper function in clusterShared.mjs to disable replication threads for specific nodes during CI test runs, and integrates it into the copyGapCursorBanking and truthResiduals integration tests. The feedback suggests wrapping the JSON.parse call within this helper in a try-catch block to handle potentially malformed environment variables gracefully and prevent test suite crashes.