fix(replication): decode-resilient outbound subscriptions + idempotent node-update watcher on deploy reload (#460)

[kriszyp]

Fixes #460

Summary

This PR closes two distinct failure modes that a deploy_component reload triggers on a follower. A deploy reloads the worker pool, which re-invokes replicator.startOnMainThread() on the same already-resolved main-thread module instance. Because whenThreadsStarted is already resolved, its .then() re-fires immediately — with no base-copy (unlike a full process restart, where hdb_nodes is base-copied first).

1. Undecodable peers dropped → zero outbound subscriptions (the data-loss symptom). The post-reload hdb_nodes subscription scan runs while a follower's replicated rows still fail to decode (a v5-era msgpackr shared-structure row misreading — the harper-pro#352 / harper#1163 family). The scan did if (!node) continue, silently skipping every undecodable peer, so the follower built zero outbound replication subscriptions (database_sockets: []) and silently stopped receiving all replicated writes — cache clears, data updates — until a full process restart. The #352 { name } fallback in resolveNodeForAuth is auth-only/inbound and is never reached by this outbound path.

2. Watcher / listener accumulation across deploys (the leak). That same re-fired whenThreadsStarted.then() re-ran subscribeToNodeUpdates → runNodeUpdateWatcher without tearing down the prior one, stacking a second unbounded watcher loop (and double-processing every hdb_nodes event), plus re-pushing routes, on every deploy. This is the source of the MaxListenersExceededWarning seen during the incident.

Fix

Decode-resilient subscription building (#1). Mirror the #352 auth-path fallback on the outbound path: when an hdb_nodes key is range-visible but its scan value is nullish, reconstruct a minimal descriptor from the key and still drive the subscription listener. Applied to both the startup/reload scan (subscribeToNodeUpdates) and the put-event path (processNodeUpdateEvent). The scan loop was extracted into pure, store-injectable helpers (scanNodesForSubscription / resolveScannedNode / reconstructNodeFromKey) for unit-testability.

Crucially, the scan does not reconstruct a genuine remove_node tombstone — a deleted node leaves a row whose POINT lookup returns null cleanly, whereas a decode failure throws from the point lookup. probeNodeRow disambiguates the two for any nullish scan value (clean null ⇒ deleted ⇒ skip; throw or present value ⇒ decode failure ⇒ reconstruct), so a removed peer is never revived/resubscribed. (The put-event path doesn't need this — genuine deletes arrive as delete events routed through isGenuineNodeDeletion, not as null-valued puts.)

Key detail: the reconstructed descriptor is { name, replicates: true }, not the auth path's { name }. The outbound path gates on node.replicates (see shouldReplicateFromNode and the early-return in subscriptionManager.onNodeUpdate); a bare { name } would be filtered out and no subscription created. replicates: true is the cluster default (what ensureThisNode and route-defaulting write), and getNodeURL already falls back to wss://<name>:9933. The next decodable hdb_nodes update replaces it with the full record.

Reload idempotency / listener-leak (#2). Make runNodeUpdateWatcher supersede its own prior instance instead of stacking one. A per-purpose generation token (watcherGenerations keyed by a string) is bumped on each (re)start; the prior loop's async iterator is closed via .return() and the older loop exits on its next turn. The key namespaces watchers so the three legitimate concurrent consumers — subscription management ('subscription-manager'), CA monitoring ('ca-monitor'), and replication-confirmation tracking ('replication-confirmation') — keep independent watchers and never supersede each other; only a re-invocation of the same consumer (i.e. a deploy reload) supersedes. subscribeToNodeUpdates gains an optional watcherKey to route each caller to a distinct key. A new exported stopNodeUpdateWatcher(key) makes the teardown explicit and testable. Separately, startOnMainThread's whenThreadsStarted.then() now clears the module-level routes list before repopulating so routes don't accumulate across deploys.

Where to look

• reconstructNodeFromKey returning replicates: true is load-bearing — please sanity-check that this does not create a subscription to a peer that shouldn't be replicated from. A reconstructed descriptor only arises for a peer whose key is already in hdb_nodes (a known, range-visible peer), and the subsequent decodable update reconciles replicates. But this is the spot to scrutinize.
• The watcher-key namespacing in runNodeUpdateWatcher / subscribeToNodeUpdates — there are exactly three call sites (subscription-manager, ca-monitor, replication-confirmation), each with a distinct key; collapsing same-key re-invocations is the intended semantics. A subscribe() that races a supersede is guarded (the loop re-checks the generation after every await), but it's worth a second pair of eyes on the await points.
• The processNodeUpdateEvent put-path: a genuine delete still routes through isGenuineNodeDeletion (unchanged); only a non-deletion null-value now reconstructs-and-forwards instead of being dropped.

What I deliberately did NOT touch

• nodeMap / connectionReplicationMap are not cleared on reload. Re-driving the same peers through onNodeUpdate is effectively idempotent (existing connections are reused via connectionReplicationMap.get), and clearing them would tear down live replication connections — too risky for a patch. The watcher-dedup + routes reset addresses the actual unbounded accumulation.
• The [DeployLifecycle] deploy:start/deploy:end listeners are added/removed by Scope (core), which already removes them in Scope.close(). The replicator uses the old startOnMainThread extension API, not a Scope, so it does not add those listeners. If the MaxListenersExceededWarning on [DeployLifecycle] persists after this fix, the residual is a Scope-close gap in core, tracked separately (see follow-ups).

Deferred follow-ups

• Periodic self-heal re-scan — defense-in-depth so a follower that somehow lands with empty subscriptions re-scans on a timer rather than only on the next hdb_nodes event.
• RecordEncoder structure-cache warm / targeted hdb_nodes re-copy at reload — the decode-failure root cause (the 5.1 line (5.0.x unaffected): rolling v4→v5 in-place upgrade — late-flipped nodes reject all inbound replication with 1008; auth-path read of v5-era hdb_nodes rows fails (exposed by #345 guard; needs new fix on main) #352 / harper#1307–#1308 family); out of scope here.
• Scope/DeployLifecycle listener audit in core — verify Scopes are reliably closed on component reload so deploy:start/deploy:end listeners don't accumulate independently of replication.

Tests

unitTests/replication/scanNodesForSubscription.test.mjs:

• Pure seams (reconstructNodeFromKey, resolveScannedNode, probeNodeRow) and the scan loop (scanNodesForSubscription): an all-undecodable follower drives zero subscriptions under the old continue skip and one per range-visible peer under the fix; a genuine tombstone is skipped (not revived) while a decode-miss peer in the same scan is still reconstructed.
• Watcher idempotency: re-invoking the same key supersedes (closes) the prior watcher; distinct keys run concurrently without superseding each other; stopNodeUpdateWatcher closes the active watcher and halts delivery.

Commands run:

• npm run build
• npx mocha --require unitTests/unitTestSetup.cjs 'unitTests/replication/**/*.test.mjs' → 172 passing (includes the existing nodeUpdateWatcher.test.mjs, unaffected by the runNodeUpdateWatcher refactor).

---

🤖 Generated with Claude Code (model: Claude Opus 4.8)

[gemini-code-assist]

Code Review

This pull request introduces generation tokens to prevent duplicate node-update watchers and implements a fallback mechanism to reconstruct minimal node descriptors from keys when decoding fails, resolving outbound subscription issues on component reloads. The review feedback suggests enriching these reconstructed nodes with properties from previous node states (oldNode) during event processing and subscription scans to prevent transient loss of properties like shard and url. Additionally, it recommends handling potential promise rejections when closing the async iterator in stopNodeUpdateWatcher to avoid unhandled promise rejections.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[claude]

Reviewed; no blockers found.