replication: recover outbound subscription wedged connected:false after peer restart (#466)

[kriszyp]

Fixes #466.

Summary

A node reconnecting to a freshly-restarted peer could wedge connected:false permanently — no socket, no SYN, no scheduled retry timer, and findWedgedNodeUrls/reconcileWorkers never re-driving it. This fixes the two recovery gaps that were code-confirmed in v5.1.10 (the third, watchdog-rearm, is a noted follow-up — see below).

The two fixes

PRIMARY — connect() reschedules on a createWebSocket rejection.
connect() is re-driven via setTimeout(() => this.connect()) from the close handler and forceReconnect() with no .catch(). connect() awaits createWebSocket(), which can reject before any socket / open / error / close listener exists — the no-valid-cert throw, or SNICallback.initialize() rejecting while a freshly-restarted peer rebuilds its TLS secure contexts. The old finally cleared reconnectScheduled and let the rejection escape as an unhandled rejection: no socket, no timer, connected:false forever.

connect() now catches that rejection and funnels it into a shared scheduleReconnect() (extracted from the close handler; forceReconnect() uses it too). The flag stays true with a fresh backoff retry, and the connection self-heals once the peer's certs settle. reconnectScheduled is cleared only on the success path (after this.socket is assigned — the original #420 semantic). The two unguarded connection.connect() call sites in replicator.ts (initial subscribe) are covered by the same internal handling.

BACKSTOP — findWedgedNodeUrls catches the never-connected entry, and the re-drive actually reconnects it.
The predicate required entry.disconnectedAt != null, which only disconnectedFromNode stamps (close / forceReconnect). A connect() that never fired 'open' leaves the entry connected:undefined with no disconnectedAt, invisible to the 30s wedge reconcile. The entry now records createdAt at creation; the predicate uses connected !== true and downSince = disconnectedAt ?? createdAt, and the reconcile re-drive loop mirrors it (connected === true skip).

The reconcile's re-driven subscribe-to-node also now carries forceReconnect:true. subscribeToNode reuses the cached connection when isReusableConnection is true, and subscribe() alone only re-emits the listener — so a never-opened (still-reusable) connection would otherwise be re-subscribed and stay wedged. forceReconnect() drives an independent reconnect and no-ops when a retry is already pending. (The original #233/#289 wedge healed without this only because that case is intentionallyUnsubscribed, hence non-reusable.) Two guards keep this from misfiring (both from cross-model review): the force only applies to a reused connection (a freshly-created one already called connect() — forcing it mid-await could open a duplicate socket), and the reconcile loop re-checks the full findWedgedNodeUrls predicate per database (a peer URL is flagged because some db is wedged, but a sibling db may be healthy or still in its initial-connect grace period).

Where to look / what to verify

• Backstop false-positive guard (the main risk): the widened predicate and the forceReconnect:true re-drive must not churn a connection that is legitimately mid-initial-connect or a healthy slow base copy. A healthy connection flips connected:true via connectedToNode on 'open' within seconds, and a base copy runs while connected:true, so neither is flagged. The reconcile re-drive re-applies the same per-db predicate (connected !== true + downSince = disconnectedAt ?? createdAt past threshold + still desired), and only force-reconnects a reused connection — so a fresh sibling db or an in-flight dial is left alone. The createdAt fallback only matters for an entry that never reached 'open', still threshold-gated by a real timestamp.
• reconnectScheduled state machine: the success path clears the flag after socket assignment; the reject path leaves it true with a pending timer; neither leaves it stuck-true-without-retry nor false-without-retry. The close handler now sets the flag (it previously did not) — this is more aligned with the existing #420 anti-double-arm intent, not less.

Deferred follow-up (NOT in this PR)

The receive watchdog is stop()-ed under backpressure (addPauseReason), so a leg that dies while paused at high backpressure won't fire forceReconnect() until resume. Re-arming it (or a separate liveness check not suppressed by pauseReasons) under sustained backpressure is a separate change. The two fixes here already break the permanent wedge; that would tighten the base-copy-stall case further.

Tests

• unitTests/replication/connectReschedulesOnRejection.test.mjs (new): a createWebSocket rejection reschedules (no unhandled rejection / permanent stuck), reconnectScheduled ends consistent with a pending retry, the armed retry actually fires, and an intentionally-unsubscribed connection does not revive. Verified failing on the pre-fix build, passing after.
• unitTests/replication/findWedgedNodeUrls.test.mjs (extended): flags a never-connected entry via createdAt past threshold; does not flag a fresh entry or one whose recent disconnectedAt should win over a stale createdAt. The never-connected case fails pre-fix, passes after.
• unitTests/replication/forceReconnect.test.mjs (extended): forceReconnect() drives recovery on a never-opened, still-reusable connection (no socket yet).
• Full unit suite green (179 passing).

Note: the wiring of forceReconnect:true through the worker subscribe-to-node message into subscribeToNode is exercised at the primitive level (the forceReconnect test above) but not end-to-end in a unit test — the worker's connection cache (connections) is module-scoped with no test seam, and a full reconnect-after-restart integration test is macOS-loopback-flaky in the harness (the same secure TLS issue noted in #466). It's a one-line if (request.forceReconnect) conditional.

🤖 Generated with Claude Code (model: Claude Opus 4.8)

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[claude]

Reviewed; no blockers found.

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[kriszyp]

Dependency note: the reconnect-recovery backstop this PR arms runs behind shouldReplicateFromNode (isDesired gate in findWedgedNodeUrls/reconcileWorkers). On a repeatedly-upgraded node that predicate's self-record point-lookup can be #352-undecodable → returns falsy → the backstop never sees the wedged peer. So this backstop is effectively a no-op for the exact field wedge it targets until #474 (decode-resilient self-record gate) lands. Recommend sequencing/landing #474 with or before this. CDP-confirmed on the field cluster: stuck connection had isFinished:true, intentionallyUnsubscribed:true, retries:0, with the self-record decode warning present.