RocksDB receiver silently drops records during bulk full-table copy (lmdb→rocksdb migration data loss) #308

Summary

During a v4→v5 migration full-table copy (and any high-rate bulk ingest into a RocksDB receiver), the receiver silently drops committed records: tables freeze short of their source counts with no retry and no operator-visible error. This is the concrete lmdb→rocksdb data-loss risk for large-customer migrations. Reproduced twice in our internal migration test bench (the internal migration-bench repro; see the internal findings record, F2), with OOM/memory confounds ruled out (v4 OOM false, node1 ~51% of 2g).

Symptom

Mid-copy, the v5 receiver repeats an unhandledRejection in a worker thread:

Transaction commit failed: Operation failed. Try again.: Transaction could not
check for conflicts for operation at SequenceNumber 46624 as the MemTable only
contains changes newer than SequenceNumber 49969. Increasing the value of the
max_write_buffer_size_to_maintain option could reduce the frequency of this error.

Tables stop at race-dependent offsets (e.g. data.users 3,742 · data.events 2,988 · mig2.users 1,490 of 4,000 — different offsets across runs), over a healthy connection (zero disconnects), with no error surfaced to any gate or operator signal. Only strict per-table count parity catches it.

Mechanism

RocksDB's optimistic-transaction conflict check needs the txn's snapshot sequence to still be covered by the memtable's maintain-window. Under the bulk-ingest burst the window overflows (the snapshot sequence falls behind max_write_buffer_size_to_maintain), the commit is rejected with "Try again," the rejection is unhandled, and those records are lost. No such error class exists on an LMDB receiver — this is RocksDB-specific.

Same memtable/maintain-window dimension as the rocks-memtable-oom-2026-06-04 investigation (WBM A/B). Note: #255 (resumable copy), #248 (keep-alive/yield), #234 (receive watchdog) are all in 5.1.0-beta.1 already — this is current behavior on top of those.

Why it matters

Silent data loss on the migration's core mechanism. A partial copy looks like slow catch-up; an operator following the runbook gates would only catch it via per-table count parity (now an explicit gate). For our largest production datasets this is a hard blocker.

Fix options (need a decision)

1. Retry-on-conflict in the apply/copy path — catch the "Try again" rejection and retry the commit (and surface a hard error if retries exhaust). At minimum, the rejection must stop being silently unhandled.
2. Raise max_write_buffer_size_to_maintain (and/or related memtable knobs) for the receiver, at least during migration/bulk-copy.
3. Write Buffer Manager (WBM) work — coordinate memtable memory so the maintain-window doesn't overflow under burst (ties to the rocks-memtable investigation).
4. Backpressure the copy sender so ingest rate stays within the conflict-check window.

(1) is the floor — silent loss must become loud — regardless of which throughput fix is chosen.

Repro

One command on our internal migration test bench (knobs scale record counts). Full run records + the exact log lines are in the internal findings record.

Related

• internal migration-bench findings record (F2)
• rocks-memtable-oom-2026-06-04 investigation (same maintain-window dimension)
• fix(replication): resumable bulk clone copy (no restart-from-zero) #255 / fix(replication): keep-alive by byte activity + decode-loop yield budget #248 / fix(replication): arm client-side receive watchdog to catch silently-dead WS #234 (in beta.1; this persists on top of them)
• Likely touches @harperfast/rocksdb-js and the harper replication receive/apply path.

🤖 Filed by Claude Opus 4.7 on Kris's behalf; findings by Nathan's internal migration-bench run.