Skip to content

createAuditEntry shared ENTRY_HEADER view races on no-encodedRecord path, corrupts txnlog #1132

Description

@ldt1996

Summary

createAuditEntry returns a view into the module-level singleton ENTRY_HEADER buffer on the no-encodedRecord return path. The next createAuditEntry call writes into ENTRY_HEADER from byte 0 again and overwrites the previous return value's underlying bytes before rocksdb-js's log.addEntry necessarily captures them. The result is corrupt entries on disk; a later read fails with RangeError: Corrupt transaction log entry at position N: declared length L overruns the log, killing whichever WS handler triggered the read and cascading into a 1006 reconnect storm.

Where

resources/auditStore.ts:397-400:

const header = ENTRY_HEADER.subarray(0, position);
if (encodedRecord) {
    return Buffer.concat([header, encodedRecord]);   // safe (concat copies)
} else return header;                                 // shared-buffer view

ENTRY_HEADER is a 2816-byte module-level singleton at resources/auditStore.ts:56. Every createAuditEntry call writes into it starting at start (default 0) and the no-encodedRecord path returns a subarray view, not a copy.

When it manifests

The bug is engine/version-independent but only fires when:

  1. The no-encodedRecord return path is hit (deletes, invalidates, any audit entry that carries no record payload).
  2. Two createAuditEntry calls happen close enough in time that the second one mutates ENTRY_HEADER before the first return value has been copied into native storage by log.addEntry.

Observed in production on a v4→v5 stage cluster at ~28 MB/min inbound system-db replication from 3 v4 peers (dominated by analytics.replicate: true, matching the rt-ite analytics-storm pattern). v4→v5 also produces a delete-heavy initial sync mix that exercises the no-encodedRecord path. v5↔v5 clusters with comparable sustained write pressure and delete share would also corrupt.

Reproducer

The corrupt file from the stage cluster is at:

/home/harperdb/harper/database/system/transaction_logs/<peer>/14.txnlog

Read fails with two error variants at the same position 0x95 depending on buffer fill level:

RangeError: Corrupt transaction log entry at position 95 of log 14: declared length 177 overruns the log (limit=195)
RangeError: Corrupt transaction log entry at position 95 of log 14: declared length 97 overruns the log (limit=16777115)

Stack:

transaction-log-reader.ts:191 (rocksdb-js native)
RocksTransactionLogStore.ts:277 (merge iterator next())
notifyFromTransactionData (transactionBroadcast.ts:164)

The uncaught throw kills the WS handler, the receiver requests full copy on reconnect, replay re-hits the same corrupt entry, loop.

Fix

Two-part:

  1. Writer: defensive copy on the no-encodedRecord return path. Change return header to return Buffer.from(header) at resources/auditStore.ts:400. Eliminates the race.
  2. Reader: wrap the iterator.next() calls in RocksTransactionLogStore's merge iterator with try/catch on RangeError; log and mark that log done so the merge keeps draining the others. Prevents one corrupt entry from cascading into a 1006 storm on any future corruption.

PR linking once opened.

Activity

  1. self-assigned this
    on Jun 4, 2026
  2. added this to the v5.0 milestone on Jun 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Fields

Priority

None yet

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions