Record what a worker-exit truth stamp did not land, and which writer saw it

[kriszyp]

main went red at 16377b8a6 on one job — Cluster Integration Tests 1/6 (Node.js v26.5.0) — and re-running it did not clear it. The failing test is truthResiduals.test.mjs:201 R1, which kills the HTTP worker that owns a subscription and waits for cluster_status to report that link down with the worker-exit code 100001. It reports the link down, with no code at all.

Chasing it, the thing that was missing was not a fix but a record. The main thread decides, at exactly one moment, whether to stamp that code — and when it declines, nothing anywhere says what it saw. This PR adds that record. It is not the fix for that failure — the root cause was found in parallel and is fixed elsewhere (see below); this is the observability that would have shortened getting there, and that still applies to every other way the stamp can decline.

Changes

stampWorkerExitTruth discarded stampWorkerExitDown's result, so a declined stamp was invisible. It now returns a reason, and every way of not stamping names itself — 'no peer name', 'no audit store', 'threw', or the buffer's own slots. That distinction is load-bearing: silence has to mean "stamped" and nothing else, or an operator reading no line during a shutdown concludes a stamp landed that never did.

What a refusal reports is raw slots, not a verdict. Any predicate over the close code would be wrong in both directions — the watchdog teardown writes CONNECTION_STATE_DOWN with no code, and no reconnect path clears the code, so a stale one survives into the next session. CONNECTION_STATE_DOWN is also 0, the value an untouched buffer already holds, so only the error time separates "an owner wrote here" from "no owner ever reached this buffer" — which is why the line carries ages rather than epoch stamps.

stampWorkerExitDown has two callers, and both now report. The reconcile sweep is the backstop for a worker dropped from the pool without ever firing exit — precisely the case where the exit handler produces nothing — so covering only the handler would have left the gap the next investigation walks into. The sweep's refusals are latched per (database, peer) like the reportedNonMemberStatus set beside them, because a dead owner is a level rather than an edge and a 5s sweep would otherwise repeat it forever.

Both go through one contained reporter that names its writer, at debug, one line per event: a refusal is usually benign, neither caller has an outer catch, and a worker owning many (database, peer) pairs would otherwise emit one line each through a crash loop.

What actually broke main — corrected

I originally CI-bisected this to harper#2524 (feat(threads): run an isolated application in a dedicated worker thread) on workflow_dispatch branches with only the core pointer moved. That attribution is wrong and is withdrawn. It was root-caused in parallel by task:main-red-kriszyp_harper-pro_056ecfd01, reproduced locally, and fixed on branch fix/replication-shared-status-gc-eviction.

The real cause is buffer lifetime, and it is present on every head since harper-pro#814 — the head-specificity my bisect saw was runner load. The per-(database, peer) shared-status buffer is engine-owned memory, and @harperfast/rocksdb-js documents that "once all ArrayBuffer instances have gone out of scope and garbage collected, the underlying memory and notify callback will be freed". getReplicationSharedStatus builds a throwaway Float64Array on every call and nothing retains one, so after the owning HTTP worker's thread is gone the allocation is reclaimed and the next resolution hands back freshly zeroed memory. The stamp is written correctly and then vanishes. That is also why it is RocksDB-only: lmdb-js keeps user buffers in a permanent per-env map and never frees them.

My bisect was not a controlled experiment, and it is worth saying why so the next person does not repeat it: the "fails with #2524, passes without" split rested on a single green run at the pre-#2524 pointer, against a failure that depends on GC timing; and the rate I quoted compared five days of main push runs against an hour of my own back-to-back dispatches on a loaded shared runner pool. Two incomparable populations, one lucky run.

What survives from that investigation is the evidence in this PR's favour, not the attribution: the main thread's own reconcile logged state=0, liveness 1388ms ago, last close code 100001 while a read moments later saw all 32 slots zero. That contradiction is what a refused-stamp record makes visible without a local reproduction.

For the human reviewer

• Is this still worth merging? Reasonable people could say no. The specific question it was built to answer has since been answered by fix/replication-shared-status-gc-eviction, and this is modest observability on a path that will be less mysterious once that lands. I think the durable half stands on its own — silence from the exit handler must mean "stamped" and cannot also mean "the database was torn down and nobody looked" — but closing this as redundant is a legitimate call and I would not argue it.
• Check for overlap before merging. fix/replication-shared-status-gc-eviction fixes the buffer lifetime in replication/knownNodes.ts; this touches subscriptionManager.ts and replicationConnection.ts, so they should be independent, but they are the same subsystem and that branch landed while this one was in review.
• Declined a review finding: that the sweep's if (!nodeName) continue / if (!auditStore) continue should report too. Those skip the whole entry — the non-member clearing, the truth derivation, the metrics bridge — not just the stamp, so instrumenting them is a different and wider diagnostic with its own latch and noise profile, and on a tick that hits them far more than the stamp is already lost. Left as-is.
• Declined too: the logger.trace?. in stampWorkerExitTruth's own catch can throw and escape. It predates this branch and is the same shape as ~87 other unguarded logger.*?. calls across replication/; guarding one of them is inconsistent, guarding all is a different PR.
• reportedUnstampedDeadOwner has no removal path for a dropped database, so the set can retain (db, peer) strings for process lifetime. Kept as-is: the pre-existing reportedNonMemberStatus beside it has the identical shape, and fixing one without the other would be worse than fixing neither.
• Not exercised end-to-end. The new branches are closures inside startOnMainThread, so the unit tests reach the formatter only. In every local reproduction attempt the stamp succeeded, so nothing here proves the log emits on the case it targets.
• replication/replicationConnection.ts is not Prettier-clean on main, so the review CLI's format self-check flags it. Left alone on purpose — reformatting that line is churn in a 7600-line file that concurrent replication work also touches. CI's gate is lint:required (oxlint), which passes.

Verification

• npm run test:unit — 1071 passing (1068 on base + 3 new).
• npm run test:integration:cluster -- --shard=1/6 on Node 26.5.0, HARPER_INTEGRATION_TEST_CONCURRENCY=2 (the CI shape) — 25/25, run four times. One run showed 2 failures in connectedBitRestartChurn.test.mjs; re-run in isolation and again as a full shard with the box quiet, both green — load on a shared box, not this change.
• npm run test:integration -- integrationTests/cluster/truthResiduals.test.mjs — 4/4.
• npx oxlint --quiet . (lint:required) — clean. Prettier clean on both changed paths.
• Independent pre-push review: 5 rounds, converged, reviewers codex + gemini + cursor-grok + harper-domain.

Refs #431

_{Review-Coverage: authored=claude; ran=gemini,codex,cursor-grok; adjudicated=domain; declined=cursor-composer; rounds=5; full=2 @ f393e94}

_{Human-Review-Need: 4 (decisions: fix-trace-escape-here-vs-separate-issue, latch-cadence-for-a-refused-stamp, debug-level-for-a-non-landing-stamp, comment-density-under-iterative-review, second-formatter-vs-reuse) @ f393e94}

[gemini-code-assist]

Code Review

This pull request improves diagnostics for replication connection tracking by introducing detailed logging for worker exit stamps that fail to land. It adds a helper function describeRefusedWorkerExitStamp to format status information and uses a new reportedUnstampedDeadOwner set to prevent redundant logging during the reconciliation sweep. However, a potential memory leak was identified: keys in reportedUnstampedDeadOwner and reportedNonMemberStatus can accumulate indefinitely when nodes are deleted or databases are removed, as the cleanup path in reconcileWorkers is bypassed. Explicit cleanup during these events is recommended.