Skip to content

fix(replication): use getSync for hdb_nodes point reads (RocksDB MaybePromise) - #476

Merged
kriszyp merged 9 commits into
mainfrom
kris/get-maybepromise
Jun 24, 2026
Merged

kriszyp merged 9 commits into
mainfrom
kris/get-maybepromise

Conversation

@kriszyp

@kriszyp kriszyp commented Jun 24, 2026 •

Copy link
Copy Markdown
Member

Fixes a class of RocksDB store.get() misuse on hdb_nodes. On RocksDB store.get() returns a MaybePromise — the record synchronously when it's cached, but a Promise on a block-cache miss. An un-awaited primaryStore.get(...)?.field then silently misfires once hdb_nodes grows past the block cache (or right after a cold-cache restart): a Promise is truthy and has none of the record's fields. Same family as #474; this PR covers the other call sites.

Summary

Switch these hdb_nodes point reads to the synchronous getSync():

  • subscriptionManager.ts — the self-node replication gate (!selfNodeRow?.replicates). This is the worst one: a Promise self-row silently sets isFullyReplicating = false and logs "Disabling replication". Also ensureThisNode, ensureNode (so the revoked_certificates merge and the sticky LOCAL_ONLY/investigate: v5-leaf → v4 audit forwarding rejected by v4 cert validation #246 bit survive), and the identity-mismatch enumeration. Extracted a small readNodeRowSync helper for the self-gate.
  • knownNodes.ts — the full get()→getSync() sweep, incorporated from fix(replication): synchronous getSync for hdb_nodes point lookups (#470) #474 (which this PR supersedes): readNodeForAuth, storeRecordRangeVisible, probeNodeRow, the two isLeader-patch reads (server.nodes push / watcher-listener forward), and shouldReplicateFromNode's self-record term via the new selfNodeReplicates helper. shouldReplicateFromNode is the load-bearing predicate for shouldSubscribe and the wedge-reconcile isDesired check, so a Promise there silently unsubscribed a still-desired peer (Codex flagged this as a P1).
  • replicator.ts — getRetrievalConnectionByName (a Promise node?.url is undefined, so the retrieval connection silently never opens).
  • clusterStatus.ts — the self-record shard/url read (a Promise omits them from cluster_status).

Supersedes #474. #474 had independently converged on the same root cause (its comments now note the #352 "empty object" was "actually a pending Promise") and was itself a comprehensive knownNodes.ts getSync sweep. Its entire knownNodes.ts change + selfNodeReplicates helper + tests are folded in here verbatim, so #474 can be closed. Note: three of its sites (readNodeForAuth / storeRecordRangeVisible / probeNodeRow) were ones this PR's original scan under-classified as safe "defensive readers" — a Promise doesn't throw and is truthy, so it slips through their try/catch + existence checks. That miss is the argument for the typed-Store<V> follow-up, which turns this whole class into compile errors. Since main is the 5.1.x line, merging here serves the patch directly (no separate patch branch needed).

Tests

  • unitTests/replication/readNodeRowSync.test.mjs — deterministic; models the RocksDB MaybePromise contract with a rocksLikeStore stub (same pattern as fix(replication): synchronous getSync for hdb_nodes point lookups (#470) #474's selfNodeReplicates test). Red-before/green-after.
  • integrationTests/cluster/blockCacheEviction.test.mjs — 3-node RocksDB cluster with a 1 MB block cache, a cold rolling restart, and a remove_node/add_node cycle; asserts the post-restart write converges, cluster_status reports the self record, and no node logs "Disabling replication".

Where to look

  • Uniform getSync (not await) is a deliberate minimal-diff choice — it restores the prior LMDB-always-sync semantics; these are tiny system rows so a blocking point read is negligible. Flagging it as the main thing to sanity-check.
  • The integration test is a strong behavioral guard but, being cache-timing dependent, is not perfectly deterministic; readNodeRowSync.test.mjs is the deterministic guard.
  • Depends on the paired core PR fix: use getSync for system-table point reads (RocksDB MaybePromise) harper#1473; the core submodule pointer is bumped to it.

Generated by Claude Code (Claude Opus 4.8).

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request addresses a critical replication bug where RocksDB's store.get() returns a Promise on a cache miss, causing replication to be silently disabled or nodes to be dropped from cluster status. The fix replaces these un-awaited get() calls with synchronous getSync() calls across several replication paths (clusterStatus.ts, knownNodes.ts, replicator.ts, and subscriptionManager.ts), and introduces integration and unit tests to prevent regressions. The review feedback focuses on performance optimizations, suggesting the avoidance of redundant synchronous database reads by passing pre-read records, utilizing getRange instead of looping over keys with synchronous point reads, and parallelizing asynchronous operations in the test setup using Promise.all to speed up execution.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread replication/subscriptionManager.ts
Comment thread replication/subscriptionManager.ts
Comment thread replication/subscriptionManager.ts
Comment thread integrationTests/cluster/blockCacheEviction.test.mjs Outdated
Comment thread integrationTests/cluster/blockCacheEviction.test.mjs Outdated
Comment thread integrationTests/cluster/blockCacheEviction.test.mjs Outdated
@claude

claude Bot commented Jun 24, 2026 •

Copy link
Copy Markdown
Contributor

Reviewed; no blockers found.

@kriszyp

kriszyp commented Jun 24, 2026

Copy link
Copy Markdown
Member Author

Thanks for the review, Gemini — all six suggestions are performance / test-speed micro-optimizations, and we're intentionally not optimizing the functions this PR touches (they're cold paths handling tiny system records). Adjudication, resolving the threads:

Sync-read suggestions (declining):

  • ensureThisNode pre-read param / pass existingNode — keeping ensureThisNode parameterless for a minimal, reviewable diff. The duplicate getSync(thisName) is a single tiny hdb_nodes point read on the startup/config-change path; the cost isn't meaningful here.
  • getRange({}) instead of N getSync in the identity-mismatch enumeration — that branch only runs when this node's self row is absent (a rare diagnostic), so the per-node point reads aren't a concern. getRange is a fair future cleanup, but out of scope for this fix.

Test-parallelization suggestions (declining):

  • Sequential setup (seeding, add_node, waitForCount) is intentional for readability and determinism. In particular, adding nodes B and C serially avoids racing two subscribers into the same leader in a test that asserts convergence — test wall-clock isn't a concern here.

None of these affect correctness, so I'm resolving the threads. Appreciate the thoroughness!

— KrAIs (Claude Opus 4.8)

@kriszyp
kriszyp marked this pull request as ready for review June 24, 2026 13:10
@kriszyp
kriszyp requested a review from a team as a code owner June 24, 2026 13:10
Comment thread integrationTests/cluster/blockCacheEviction.test.mjs Outdated
Comment thread integrationTests/cluster/blockCacheEviction.test.mjs Outdated
kriszyp and others added 9 commits June 24, 2026 07:57
…ePromise)

On RocksDB store.get() returns a Promise on a block-cache miss; the self-node
replication gate, ensureThisNode/ensureNode, the retrieval-connection lookup,
cluster_status, and the node-update watcher consumed that result synchronously,
so once hdb_nodes grew past the block cache (or right after a cold-cache restart)
a Promise self-row silently disabled replication, dropped peers from
cluster_status, or skipped opening a connection. Switch those reads to getSync
(extracted readNodeRowSync helper for the self-gate). Leaves shouldReplicateFromNode
to PR #474. Adds a deterministic unit test (readNodeRowSync) and a cluster
integration test (small block cache + cold restart + remove/add node). Bumps the
core submodule to the paired system-table getSync fix.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…x P1)

shouldReplicateFromNode is the load-bearing predicate for BOTH shouldSubscribe
(onDatabase) and the wedge-reconcile isDesired check. Its final self-record term
still used an un-awaited primaryStore.get(getThisNodeName())?.replicates, so a
RocksDB block-cache miss (cold restart / grown hdb_nodes) made the whole predicate
falsy and silently unsubscribed a still-desired peer — upstream of the readNodeRowSync
guard added earlier in this branch. Switch it to getSync. PR #474 refactors this same
expression to selfNodeReplicates, which must likewise read sync.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…s PR

Supersedes #474. Adopts its comprehensive knownNodes.ts get()->getSync fix verbatim,
including readNodeForAuth / storeRecordRangeVisible / probeNodeRow — sites this branch's
original scan had under-classified as safe '#352 defensive readers' (a Promise from get()
does not throw and is truthy, so it slips through their try/catch + existence checks and
strands the peer / makes an absent key look present / revives a tombstone). Brings #474's
selfNodeReplicates helper + its unit test and the updated readNodeForAuth and
scanNodesForSubscription tests. shouldReplicateFromNode now reads via selfNodeReplicates
(getSync), replacing this branch's earlier inline getSync there.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…odes, sentinel convergence)

The 1MB blockCacheSize + 512KB WriteBufferManager (allowStall) was sub-viable for
opening Harper's RocksDB databases — every node hung at startup (HarperStartupError,
no output for 60s), failing the suite. Use a small-but-viable 32MB block cache and
drop the WBM override; the COLD RESTART is the deterministic out-of-cache trigger
(the field repro). Also trimmed to 2 nodes and switched convergence checks to sentinel
records (no fragile full-count/pagination) for speed and reliability.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…CONNREFUSED

remove_node reloads the node, so its operations API briefly refuses connections;
the immediate add_node hit ECONNREFUSED (TypeError: fetch failed). Wait for the node
to be reachable (pollHealth) and retry the re-add. (Cold-restart test 1 already passes.)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…cheEviction

Test 2 failed on convergence after removing a node's leader and re-adding: the node
ends up with a null self-record and (correctly) disables replication, not re-converging
in-window. The log shows selfNodeRow=null (a concrete value, NOT a Promise) — so this is
a remove_node re-subscription behavior orthogonal to the get() MaybePromise fix this suite
guards. The cold-restart test already exercises ensureThisNode/shouldReplicateFromNode/
cluster_status on a cold cache; add_node is covered by the before hook and
replicationReconnect/replicationTopology tests.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
…rror

Adds a NodeStore type whose get() returns MaybePromise<NodeRecord> (the honest RocksDB
contract — a Promise on a block-cache miss) and types getHDBNodeTable()'s return, so any
SYNCHRONOUS consumption of get() (e.g. store.get(id)?.replicates) is now a TypeScript
error — use getSync() or await/when. The value type reuses the canonical Node type plus
an index signature for incidental fields (e.g. authorization).

tsc is clean against this branch with the typing applied — proving the get->getSync fixes
already on this branch leave ZERO remaining sync misuse on hdb_nodes — and a deliberate
store.get(x)?.shard is correctly rejected. This is the compiler-driven backstop for the
store.get() MaybePromise class (would have caught the readNodeForAuth/probeNodeRow misses
the manual audit under-classified). Pairs with the rocksdb-js DBI<T,V> value-generic.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant