Skip to content

fix(table): bound out-of-order audit-chain walk to prevent full-copy OOM - #1116

Merged
kriszyp merged 1 commit into
mainfrom
kris/fix-1114-bounded-audit-walk
Jun 3, 2026
Merged

kriszyp merged 1 commit into
mainfrom
kris/fix-1114-bounded-audit-walk

Conversation

@kriszyp

@kriszyp kriszyp commented Jun 3, 2026

Copy link
Copy Markdown
Member

Summary

Bounds the out-of-order-write reconciliation walk in Table.commit so a deep audit history can't pin the JS heap during replication full-copy catch-up.

  • Materialize-and-drop: the backward audit-chain walk now retains only { version, value } per succeeding update instead of the full audit record, releasing each entry's backing transaction-log buffer + decoders immediately. This is semantics-preserving and is the main heap reduction.
  • Depth cap (RocksDB only): beyond MAX_OUT_OF_ORDER_AUDIT_DEPTH (1000) walk steps the walk stops and reconciles the older write against the retained most-recent window via the existing rebuildUpdateBefore fold. A single O(1) lookup at txnTime rules out re-applying a duplicate (full-copy audit-replay re-delivers writes) so commutative ops aren't double-applied. LMDB keeps its exact, unbounded walk (its audit entries are keyed by local audit time, so the duplicate shortcut wouldn't apply, and the OOM was RocksDB-only).

Purpose

Fixes the OOM in #1114: during full-copy of a large-history database, the reconciliation walked the entire backward chain for every record, synchronously, on all workers — peak heap scaled with audit depth × worker count and the cgroup OOM-killer crash-looped the container.

Where to put attention

  • resources/Table.ts capped branch — the bounded path is an approximation for out-of-order chains deeper than the cap: updates older than the retained 1000-window are not layered into the fold, so a stale field from the delayed write could briefly win over a newer value until the authoritative full-copy record converges it. This is the deliberate trade for a hard heap bound (discussed and chosen over a memory-only mitigation). For normal (shallow) out-of-order writes and for LMDB, behavior is unchanged. Worth a sanity check that this trade is acceptable for the non-full-copy out-of-order case.
  • The duplicate check assumes RocksDB per-node audit logs are keyed by version (auditStore.get(txnTime, …, options?.nodeId)); it only runs on the RocksDB-scoped capped path.

Not covered here

The issue also notes unbounded growth of the system-DB per-peer replication log (+ an invalid user role found storm) as a contributing factor. That's deferred to a separate investigation — this PR only makes the audit-walk memory-safe.


Tests: 2 regression tests in unitTests/resources/transaction.test.js (bounded reconciliation correctness incl. cap engagement, and no-double-apply on duplicate re-delivery). test:unit:resources green. Integration suite left to CI (a git worktree without local node_modules blocks the component-sandbox tests locally).

🤖 Generated by Claude (Opus 4.7) via Claude Code.

During replication full-copy catch-up, Table.commit's out-of-order
reconciliation walked the entire backward audit-history chain per record,
synchronously, buffering intermediate records in the JS heap. Peak heap scaled
with audit-history depth times worker count and OOM-killed the worker.

- Materialize each patch value during the walk and retain only { version, value }
  so the backing transaction-log buffer and decoders are released immediately,
  instead of pinning the full audit records for the whole walk.
- Cap the walk at MAX_OUT_OF_ORDER_AUDIT_DEPTH (RocksDB only, where the OOM was
  observed). Beyond it, reconcile against the retained most-recent window via the
  existing rebuildUpdateBefore fold and rule out duplicate re-delivery with an
  O(1) lookup at txnTime so commutative ops are not double-applied. LMDB keeps its
  exact unbounded walk.

Refs #1114

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request addresses issue #1114 by capping the out-of-order write reconciliation audit-chain walk for RocksDB to prevent worker OOM errors on pathologically deep audit histories. It also optimizes memory usage by materializing patch values immediately rather than retaining full audit records. The reviewer recommended changing the comparison operator in the duplicate check from === 0 to <= 0 to correctly skip both exact duplicates and losing concurrent writes at the same timestamp.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread resources/Table.ts
@claude

This comment has been minimized.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant