Summary
A RocksDB transaction log's keys are claimed from a process-wide monotonic clock that is not persisted across restarts: rocksdb-js docs/transaction-log.md states that "a restart after the wall clock moved backwards can reissue" a key. Replication resume filters the origin's log by key (replication/replicationConnection.ts sender: getRange({ start: cursor, exclusiveStart: true, ... })), so after such a restart every entry the origin appends with a key below a subscriber's persisted cursor is never delivered to that subscriber on reconnect. The live tail (no reconnect) is unaffected: a per-log range read yields entries in file/append order, filtered by key — verified empirically on rocksdb-js 2.9.0 with out-of-order timestamps 100, 200, 101, 300: start>50 yields 100, 200, 101, 300; start>150 yields 200, 300 (the post-rollback 101 is skipped).
Consequence: a peer that reconnects after an origin's clock rollback silently misses that origin's post-rollback writes until the origin's clock passes the peer's cursor again. Nothing logs it; the peer's status shows a healthy, advancing cursor.
Proposed fix (rocksdb-js, small)
Persist a clock floor: on open, the monotonic timestamp source starts at max(now, highest key in the local log + 1 tick) (the running-maxima index the reader already builds has that value). Keys then never regress across restarts, replay-by-key equals append order, and cursors stay sound. The alternative — an incarnation-qualified append sequence carried on the wire — fixes the same thing at the cost of a wire/storage migration and is not proposed.
Why it matters now
harper-pro#822 (cluster record locks) proves successor freshness with a lockBarrier entry (harper#2625) the origin appends after the fenced write; that proof holds on the live path and fails closed on resume (the barrier is simply never delivered → 503), so locks are not made unsafe by this — but ordinary replication is silently incomplete in the same window, which is the larger problem.
Not in scope
harper-pro's lock transport (harper-pro#822). Deduplication of reissued keys (the existing (timestamp, origin) identity rule stands).
Summary
A RocksDB transaction log's keys are claimed from a process-wide monotonic clock that is not persisted across restarts: rocksdb-js
docs/transaction-log.mdstates that "a restart after the wall clock moved backwards can reissue" a key. Replication resume filters the origin's log by key (replication/replicationConnection.tssender:getRange({ start: cursor, exclusiveStart: true, ... })), so after such a restart every entry the origin appends with a key below a subscriber's persisted cursor is never delivered to that subscriber on reconnect. The live tail (no reconnect) is unaffected: a per-log range read yields entries in file/append order, filtered by key — verified empirically on rocksdb-js 2.9.0 with out-of-order timestamps100, 200, 101, 300:start>50yields100, 200, 101, 300;start>150yields200, 300(the post-rollback101is skipped).Consequence: a peer that reconnects after an origin's clock rollback silently misses that origin's post-rollback writes until the origin's clock passes the peer's cursor again. Nothing logs it; the peer's status shows a healthy, advancing cursor.
Proposed fix (rocksdb-js, small)
Persist a clock floor: on open, the monotonic timestamp source starts at
max(now, highest key in the local log + 1 tick)(the running-maxima index the reader already builds has that value). Keys then never regress across restarts, replay-by-key equals append order, and cursors stay sound. The alternative — an incarnation-qualified append sequence carried on the wire — fixes the same thing at the cost of a wire/storage migration and is not proposed.Why it matters now
harper-pro#822 (cluster record locks) proves successor freshness with a
lockBarrierentry (harper#2625) the origin appends after the fenced write; that proof holds on the live path and fails closed on resume (the barrier is simply never delivered → 503), so locks are not made unsafe by this — but ordinary replication is silently incomplete in the same window, which is the larger problem.Not in scope
harper-pro's lock transport (harper-pro#822). Deduplication of reissued keys (the existing
(timestamp, origin)identity rule stands).