Prevent copy recovery stalls after blob faults

[kriszyp]

A queued replication record handler could begin a blob repair after connection teardown had already swept blobsInFlight. That orphaned receive stream had no sender or connection timer left to finish it, so it retained the blob lock and every reconnect declined repair until core's long source-idle timeout. Runtime scheduling changed how often the race appeared; this is not intrinsically a Node 22 defect.

This change aborts a newly attached blob receive when its closure no longer owns the websocket. The companion core change makes failed-save rejection a cleanup barrier, ensuring PENDING-marker completion and blob unlock cannot lag promise settlement.

The regression now runs pre-merge on Node 22, 24, and 26.5.0. It still requires faults across two advancing copy cursors, but injects them every five blob saves so the resumed repair pass reliably encounters another fault. Previously, two faults could both occur during the initial pass, leaving the strong two-resume assertion waiting despite complete data recovery.

Depends-on: HarperFast/harper#2228

Companion: HarperFast/harper#2228 — Release blob locks before failed saves settle.

Fixes #699

Issue: HarperFast/harper-pro#699 — 5.2.2 blob-gap wedge is bounded but banks zero copy progress — the copy cursor never persists during a copy body, and the watchdog's reconnect mints the next cycle's faults.

For the human reviewer

1. Keep the post-attach ownership check local to blob receive setup rather than restructuring record dispatch cancellation. This closes the proven race with a narrow blast radius; a broader cancellation design could supersede it later.
2. Classify the forced abort as the existing quiet transient connection-close error. The receive is unfinished but expected during ownership loss, so it must clean up without adding noisy error metrics.
3. Build three runtime-specific artifacts on every PR, but run only this dedicated cluster file on Node 22/24/26.5. Broad PR suites remain on Node 24 to limit cost; nightlies and main pushes retain broader runtime coverage.
4. Guard the dedicated job by both path and literal 5.2.2 blob-gap wedge is bounded but banks zero copy progress — the copy cursor never persists during a copy body, and the watchdog's reconnect mints the next cycle's faults #699 case name. This fails closed if the regression is removed or renamed, at the cost of coupling workflow configuration to its title.

Verification

• The focused regression passed locally in 15.9s with 10 injected faults, two banked reconnects, advancing resume keys [3, 16], 40/40 records, and no missing payloads.
• PR CI passed the dedicated regression on Node 22, 24, and 26.5.0. Builds and unit tests passed on all three runtimes; lint, YCSB, all non-cluster integration shards, and all cluster shards passed.
• Cluster shard 2 initially failed inside the downloaded legacy-v4 source. ordered-binary decoded a key as a fractional number and threw while converting it to BigInt, repeatedly closing the audit subscription. The production and regression changes in this PR do not run in that process. A targeted rerun passed.
• The focused receive-ownership unit suite passed 10/10 locally. npm run lint:required, Prettier, and workflow YAML parsing passed.
• Broader local suites reported unrelated existing failures: Pro unit completed with 775 passing and one file-mode failure. A clean Pro build emitted the repository's existing TypeScript errors while still producing dist.

Co-Authored-By: GPT-5 Codex noreply@openai.com

_{Review-Coverage: authored=unknown; ran=none; rounds=1 @ 5e171f2}

_{Human-Review-Need: 4 @ 5e171f2}

[gemini-code-assist]

Code Review

This pull request updates the core subproject commit to a newer version (035b2d6a8ea1898fadae4f903a3bef777a4c767d). There are no review comments, and I have no feedback to provide.

[claude]

Reviewed; no blockers found.

[heskew]

placeholder

---

Generated by Claude Code

[kriszyp]

Correction: the regression test does pin this fix

I reported earlier that the control passed and therefore the test did not discriminate. That was drawn from two samples of an intermittent race and was wrong. A third control run on Node 22 reproduced the original failure exactly.

Control = this PR with the ownership guard's condition hardcoded false (function and unit tests intact, production behavior equal to main): #741.

control run, Node 22	result
1	pass (53s)
2	pass (66s)
3	fail (362.7s)

Run 3's fingerprint against the original nightly red (32226893983):

banking test: injected=10 watchdogFires=13 bankedReconnects=3
  resumeKeys=[3,9,13,13,13,13,13,13,13,13,13,13,13,13,13]
  B records=40/40 missingReferencedPayloadIds=[14:error=fetch failed]
AssertionError: copy did not converge under the sustained fault supply:
  B records=40/40, records missing their blob payload: [14:error=fetch failed]
  (final banked cursor: 13)

Same assertion, same shape, same magnitude — 362.7s here against the original 363s, and against 61s on Node 24 and 70s on Node 26.5 in the same control run. The cursor pins at 13 and re-walks it thirteen times; watchdogFires goes 0 → 13.

So on Node 22 the base reproduces at roughly 1 in 3, matching this PR's own account ("two exact-Node-22 pre-fix stalls... a third run escaped"). The test is a genuine regression anchor; it is just probabilistic, and two green runs are not enough to conclude anything from it.

Caveat worth keeping on the record: with base failing ~1/3, three green runs on the fix side (1 CI × 3 runtimes, plus 3 local on Node 22) leave roughly a 30% chance of coincidence. The fix rests on the invariant argument — receiveBlobs is synchronous, so the post-attach check cannot be interleaved by the close handler — plus the captured lock-after-teardown diagnostic; the test corroborates rather than proves. Nightlies will accumulate the sample on main.

Closing #741 now that this is recorded.

— Claude Opus 5