fix(replication): recover ping-alive copy-stall wedges via a copy-progress watchdog (#453)

[kriszyp]

Update (follow-ups + milestone → v5.1). This PR now carries two approved follow-ups for harper-pro#460 on top of the original #453 copy-progress watchdog, and is retargeted from v5.2 to v5.1 as a patch-grade reliability fix. Paired core change: HarperFast/harper#1461 (adds the replication_copyTimeout config param) — the core submodule pointer is bumped to it here, so #1461 must merge / be released before this PR can build against released core.

(b) Separate copy-phase receive timeout (REPLICATION_COPYTIMEOUT, default 300000ms). Distinct from the #453 copy-progress watchdog. The byte-level receiveWatchdog and the client sendPing idle check both timed the receiver out after REPLICATION_PINGTIMEOUT (60s) with no exemption during copy. On the initial copy of a large table (e.g. hdb_analytics, ~2.75M rows) the sender sits in writableNeedDrain backpressure, so no bytes reach the receiver for well over 60s even though the copy is healthy → the watchdog fires → forceReconnect → resume the same checkpoint → loop, starving the copy. Fix: while inCopyMode the byte watchdog uses the wider COPY_TIMEOUT threshold instead of PING_TIMEOUT. createReceiveWatchdog now accepts intervalMs as a number or a function resolved per-arm, so the threshold tracks inCopyMode over the connection's life.

• Where to look: the synchronous ws.on('message') resetPingTimer() arms the byte watchdog before onWSMessage flips inCopyMode, so the byte watchdog is explicitly re-armed on COPY_START (and narrowed back when the copy finishes in maybeFinishCopy) — otherwise a sender that goes silent immediately after COPY_START would still trip the 60s ping timeout. This re-arm uses stop()+reset() to bypass the per-frame reset throttle. (Found by Codex cross-model review; fixed.)

(a) connected:true / ver=0 mid-copy blind spot — assessed, no code added. findWedgedNodeUrls (subscriptionManager.ts) only re-drives connected===false, so a subscription stuck connected:true / Receiving / version 0 is invisible to it. The #453 copy-progress watchdog already recovers exactly that case — it forces a reconnect on a connected-but-stalled copy. Extending findWedgedNodeUrls to re-drive connected:true entries would risk tearing down healthy slow-but-progressing copies, which the frame-keyed watchdog avoids by design. So (a) is covered by the existing watchdog; documented here rather than coded. This is the recovery-net side of harper-pro#460 (followers' replicated hdb_nodes failing to decode after a deploy reload).

Tests: new unit case in receiveWatchdog.test.mjs pins the function-form intervalMs per-arm resolution (normal → copy-phase widen → fire). The #453 integration test now also sets copyTimeout to prove the byte watchdog does not recover during the stall — the copy-progress watchdog is the recovery path. Both pass locally (full 155-test replication unit suite green; integration test recovers in ~13s, no restart).

The original #453 write-up follows.

---

Summary

Adds a copy-progress watchdog to the replication receive path that recovers a base copy which stalls while the socket stays ping-alive — a wedge neither existing safety net catches.

Purpose (harper-pro#453)

a customer cluster preprod 5.1.7: a rolling upgrade restart interrupted the system base copy and a follower's receive subscription settled connected:true / lastReceivedStatus:"Receiving" with the copy frozen at version 0 — permanently. Because live audit replay (which carries new hdb_deployment rows) only starts after COPY_COMPLETE, replicated deploys timed out for hours with no self-heal; only a manual staggered restart cleared it.

Both existing recovery mechanisms miss it, and for the same reason — they key off connected:false:

• findWedgedNodeUrls (the Replication wedges permanently after simultaneous cluster restart (reconciler skips open-but-idle sockets) → blocks replicated deploys #420/fix(replication): recover open-but-idle wedged subscriptions via watchdog-driven reconnect #424 wedge-reconcile) only re-drives connected:false entries.
• the byte-level receiveWatchdog never fires because keepalive pings keep ws._socket.bytesRead advancing.

The new watchdog is keyed on received copy app-frames instead of socket bytes (pings are WS control frames, not 'message' events, so they can't suppress it). While in copy mode, if no copy frame arrives for the copy-stall threshold it forces the same close-independent forceReconnect the byte watchdog uses, which restarts the copy from the leader. It's armed on COPY_START and each in-copy frame, and suspended during backpressure pause / stopped on copy finish + close, so a slow-but-progressing or paused copy never trips it.

Proven

New integration test copyProgressWedgeRecovery.test.mjs reproduces the wedge deterministically via a one-shot, env-gated sender stall (maybeStallCopyForTest) that freezes the copy right after COPY_START while pings keep flowing. Ran both ways locally: without the watchdog the subscriber stays wedged and the post-stall write never replicates (test fails); with it, the copy reconnects, converges, and replication resumes — no restart.

Where to look / open questions for the reviewer

• Copy-progress stall threshold reuses REPLICATION_BLOBTIMEOUT (default 120s) — the existing "tolerate a stalled replication transfer" knob. This is the a customer cluster preprod 5.1.7: replicated deploys wedge after rolling upgrade — stalled system blob send (#450/harper#1443) blocks COPY_COMPLETE; fixes absent from 5.1.7 (needs 5.1.8) #453 frame-keyed watchdog and is separate from the new REPLICATION_COPYTIMEOUT (b) above, which governs the byte-level idle watchdog during copy. Worth a sanity check that the two thresholds (120s frame-stall vs 300s byte-idle) read sensibly together.
• Recovery is forceReconnect → restart copy. With the system base-copy gated on huge hdb_analytics (analytics.replicate:true) blocks hdb_deployment convergence → deploys fail after any copy #421 cross-version cursor a reconnect restarts the copy from scratch; it converges once one attempt completes uninterrupted. The watchdog gives it repeated clean attempts rather than silently waiting forever.
• Test hook maybeStallCopyForTest mirrors the existing armReplicationWedgeForTest pattern (env-gated, one-shot, never arms in production).

Related: #453 (incident), #460 (this PR's follow-ups + recovery-net), #420/#424/#289 (connected:false wedge family), #241/#426 (copy restart-from-zero), #451 + harper#1444 (sibling blob-timeout watchdogs — separate hardening, not this wedge). Paired core: harper#1461.

🤖 Generated by Claude (Opus 4.8). Diff and commit history are the source of truth.

---

Open item from cross-model review (for the human reviewer)

A Gemini pass flagged that a persistently slow leader (>120s between copy frames, e.g. severe disk/CPU contention scanning a table) could trip the copy-progress watchdog and reconnect. I kept the behavior deliberately: it mirrors the existing byte receiveWatchdog, which also forceReconnects on silence without backoff; the watchdog re-arms on every copy frame, so it only fires on a true ~120s gap with zero copy progress (a genuinely stalled copy); and on a same-version cluster the reconnect resumes from the persisted copy cursor, not from zero. If we'd prefer a reconnect-attempt cap / backoff here, easy to add — flagging it as a conscious tradeoff rather than an oversight.

Open item from the follow-up cross-model review (Gemini, for the human reviewer)

On (a), Gemini did not endorse "covered by the watchdog" and recommended a defense-in-depth global net: have findWedgedNodeUrls also re-drive a subscription stuck connected:true / Receiving / version 0 for, say, >10–15 min — so that if the local watchdog itself fails (software bug, event-loop block, misconfiguration) there is still a reconcile-level recovery. I deliberately left this out of this patch: it's a larger, separately-tunable hardening (re-driving connected:true entries is exactly the teardown-a-healthy-slow-copy risk this PR otherwise avoids), and out of the patch-grade scope for v5.1. Flagging it as a conscious deferral for the reviewer to weigh — a good candidate for a follow-up issue rather than a blocker here.

(Gemini's other findings — a throttle-swallowing bug on stop()+reset(), and regressions from "deleting" the consumer-less-blob skip-pause branch / sender chunk-timeout — do not apply: the reset() throttle already short-circuits on !timer so the transition re-arm is not swallowed (proven by the existing pause→resume unit test plus the new per-arm test), and this PR does not touch sendBlob or the blob-deadlock paths at all.)

[gemini-code-assist]

Code Review

This pull request introduces a copy-progress watchdog to resolve a replication wedge issue (harper-pro#453) where a stalled base copy remains frozen while keepalive pings prevent the byte-level receive watchdog from firing. The changes include adding a new watchdog keyed on application-level copy frames, integrating it into the WebSocket replication lifecycle, and adding a comprehensive integration test with a test-only fault injection hook to verify the recovery behavior. I have no feedback to provide as there are no review comments.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[claude]

Reviewed; no blockers found.