Repository navigation
fix(table): bound out-of-order audit-chain walk to prevent full-copy OOM - #1116
Conversation
During replication full-copy catch-up, Table.commit's out-of-order
reconciliation walked the entire backward audit-history chain per record,
synchronously, buffering intermediate records in the JS heap. Peak heap scaled
with audit-history depth times worker count and OOM-killed the worker.
- Materialize each patch value during the walk and retain only { version, value }
so the backing transaction-log buffer and decoders are released immediately,
instead of pinning the full audit records for the whole walk.
- Cap the walk at MAX_OUT_OF_ORDER_AUDIT_DEPTH (RocksDB only, where the OOM was
observed). Beyond it, reconcile against the retained most-recent window via the
existing rebuildUpdateBefore fold and rule out duplicate re-delivery with an
O(1) lookup at txnTime so commutative ops are not double-applied. LMDB keeps its
exact unbounded walk.
Refs #1114
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
There was a problem hiding this comment.
Code Review
This pull request addresses issue #1114 by capping the out-of-order write reconciliation audit-chain walk for RocksDB to prevent worker OOM errors on pathologically deep audit histories. It also optimizes memory usage by materializing patch values immediately rather than retaining full audit records. The reviewer recommended changing the comparison operator in the duplicate check from === 0 to <= 0 to correctly skip both exact duplicates and losing concurrent writes at the same timestamp.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
Summary
Bounds the out-of-order-write reconciliation walk in
Table.commitso a deep audit history can't pin the JS heap during replication full-copy catch-up.{ version, value }per succeeding update instead of the full audit record, releasing each entry's backing transaction-log buffer + decoders immediately. This is semantics-preserving and is the main heap reduction.MAX_OUT_OF_ORDER_AUDIT_DEPTH(1000) walk steps the walk stops and reconciles the older write against the retained most-recent window via the existingrebuildUpdateBeforefold. A single O(1) lookup attxnTimerules out re-applying a duplicate (full-copy audit-replay re-delivers writes) so commutative ops aren't double-applied. LMDB keeps its exact, unbounded walk (its audit entries are keyed by local audit time, so the duplicate shortcut wouldn't apply, and the OOM was RocksDB-only).Purpose
Fixes the OOM in #1114: during full-copy of a large-history database, the reconciliation walked the entire backward chain for every record, synchronously, on all workers — peak heap scaled with audit depth × worker count and the cgroup OOM-killer crash-looped the container.
Where to put attention
resources/Table.tscapped branch — the bounded path is an approximation for out-of-order chains deeper than the cap: updates older than the retained 1000-window are not layered into the fold, so a stale field from the delayed write could briefly win over a newer value until the authoritative full-copy record converges it. This is the deliberate trade for a hard heap bound (discussed and chosen over a memory-only mitigation). For normal (shallow) out-of-order writes and for LMDB, behavior is unchanged. Worth a sanity check that this trade is acceptable for the non-full-copy out-of-order case.auditStore.get(txnTime, …, options?.nodeId)); it only runs on the RocksDB-scoped capped path.Not covered here
The issue also notes unbounded growth of the system-DB per-peer replication log (+ an
invalid user role foundstorm) as a contributing factor. That's deferred to a separate investigation — this PR only makes the audit-walk memory-safe.Tests: 2 regression tests in
unitTests/resources/transaction.test.js(bounded reconciliation correctness incl. cap engagement, and no-double-apply on duplicate re-delivery).test:unit:resourcesgreen. Integration suite left to CI (a git worktree without localnode_modulesblocks the component-sandbox tests locally).🤖 Generated by Claude (Opus 4.7) via Claude Code.