Summary
createAuditEntry returns a view into the module-level singleton ENTRY_HEADER buffer on the no-encodedRecord return path. The next createAuditEntry call writes into ENTRY_HEADER from byte 0 again and overwrites the previous return value's underlying bytes before rocksdb-js's log.addEntry necessarily captures them. The result is corrupt entries on disk; a later read fails with RangeError: Corrupt transaction log entry at position N: declared length L overruns the log, killing whichever WS handler triggered the read and cascading into a 1006 reconnect storm.
Where
resources/auditStore.ts:397-400:
const header = ENTRY_HEADER.subarray(0, position);
if (encodedRecord) {
return Buffer.concat([header, encodedRecord]); // safe (concat copies)
} else return header; // shared-buffer view
ENTRY_HEADER is a 2816-byte module-level singleton at resources/auditStore.ts:56. Every createAuditEntry call writes into it starting at start (default 0) and the no-encodedRecord path returns a subarray view, not a copy.
When it manifests
The bug is engine/version-independent but only fires when:
- The no-
encodedRecord return path is hit (deletes, invalidates, any audit entry that carries no record payload).
- Two
createAuditEntry calls happen close enough in time that the second one mutates ENTRY_HEADER before the first return value has been copied into native storage by log.addEntry.
Observed in production on a v4→v5 stage cluster at ~28 MB/min inbound system-db replication from 3 v4 peers (dominated by analytics.replicate: true, matching the rt-ite analytics-storm pattern). v4→v5 also produces a delete-heavy initial sync mix that exercises the no-encodedRecord path. v5↔v5 clusters with comparable sustained write pressure and delete share would also corrupt.
Reproducer
The corrupt file from the stage cluster is at:
/home/harperdb/harper/database/system/transaction_logs/<peer>/14.txnlog
Read fails with two error variants at the same position 0x95 depending on buffer fill level:
RangeError: Corrupt transaction log entry at position 95 of log 14: declared length 177 overruns the log (limit=195)
RangeError: Corrupt transaction log entry at position 95 of log 14: declared length 97 overruns the log (limit=16777115)
Stack:
transaction-log-reader.ts:191 (rocksdb-js native)
RocksTransactionLogStore.ts:277 (merge iterator next())
notifyFromTransactionData (transactionBroadcast.ts:164)
The uncaught throw kills the WS handler, the receiver requests full copy on reconnect, replay re-hits the same corrupt entry, loop.
Fix
Two-part:
- Writer: defensive copy on the no-
encodedRecord return path. Change return header to return Buffer.from(header) at resources/auditStore.ts:400. Eliminates the race.
- Reader: wrap the
iterator.next() calls in RocksTransactionLogStore's merge iterator with try/catch on RangeError; log and mark that log done so the merge keeps draining the others. Prevents one corrupt entry from cascading into a 1006 storm on any future corruption.
PR linking once opened.
Summary
createAuditEntryreturns a view into the module-level singletonENTRY_HEADERbuffer on the no-encodedRecordreturn path. The nextcreateAuditEntrycall writes intoENTRY_HEADERfrom byte 0 again and overwrites the previous return value's underlying bytes before rocksdb-js'slog.addEntrynecessarily captures them. The result is corrupt entries on disk; a later read fails withRangeError: Corrupt transaction log entry at position N: declared length L overruns the log, killing whichever WS handler triggered the read and cascading into a 1006 reconnect storm.Where
resources/auditStore.ts:397-400:ENTRY_HEADERis a 2816-byte module-level singleton atresources/auditStore.ts:56. EverycreateAuditEntrycall writes into it starting atstart(default 0) and the no-encodedRecordpath returns asubarrayview, not a copy.When it manifests
The bug is engine/version-independent but only fires when:
encodedRecordreturn path is hit (deletes, invalidates, any audit entry that carries no record payload).createAuditEntrycalls happen close enough in time that the second one mutatesENTRY_HEADERbefore the first return value has been copied into native storage bylog.addEntry.Observed in production on a v4→v5 stage cluster at ~28 MB/min inbound system-db replication from 3 v4 peers (dominated by
analytics.replicate: true, matching the rt-ite analytics-storm pattern). v4→v5 also produces a delete-heavy initial sync mix that exercises the no-encodedRecordpath. v5↔v5 clusters with comparable sustained write pressure and delete share would also corrupt.Reproducer
The corrupt file from the stage cluster is at:
Read fails with two error variants at the same position 0x95 depending on buffer fill level:
Stack:
The uncaught throw kills the WS handler, the receiver requests full copy on reconnect, replay re-hits the same corrupt entry, loop.
Fix
Two-part:
encodedRecordreturn path. Changereturn headertoreturn Buffer.from(header)atresources/auditStore.ts:400. Eliminates the race.iterator.next()calls inRocksTransactionLogStore's merge iterator with try/catch onRangeError; log and mark that log done so the merge keeps draining the others. Prevents one corrupt entry from cascading into a 1006 storm on any future corruption.PR linking once opened.