Mesh replication amplifies a re-delivered hdb_nodes delete into millions of duplicate audit entries; the resulting same-version run permanently exceeds replication_maxPayload #826

Repo: harper-pro. Observed on: harper-pro 5.2.7, RocksDB storage, 16-node full-mesh production cluster ([customer-cluster]). A prior occurrence on the same cluster on 2026-08-17 (~394 MB frames) self-healed when the entries aged out of auditRetention (3d). Related to but distinct from #711.

Summary

Two defects compound:

Defect A (amplification): a replicated delete of a record that does not exist locally is re-applied and re-logged on every delivery, with no dedup. Every dedup layer requires a stored record to tie against: isDurableIdentityTie returns false when !existing (replicationConnection.ts ~1113), the leading-duplicate fast-skip calls it through readLocalEntry (~3780), and the CRDT resequencing walk with its keyed auditStore.get(txnTime, tableId, id, nodeId) dedup (Table.ts ~2602, ~2667) only runs when an existingEntry is present. A delete of an absent row hits none of these, applies as a no-op, and appends a fresh audit entry keyed at localTime = version (the apply transaction inherits the event's timestamp: Table.ts ~944, transaction.ts ~52). In a mesh, each appended copy is new log tail that every other leg picks up and forwards; each receiver re-logs it again. The result is exponential echo: one delete entry became ~6.4 million byte-identical audit entries per node in 16 minutes.

Defect B (no flush inside a same-version run): the audit send path cannot ship a run of entries sharing one version, no matter its size. sendAuditRecord flushes only when auditRecord.version changes (~4972) or on the synthesized end_txn at iterator drain (~5508). #711 added COPY_FLUSH_BYTES pacing to the bulk-copy path; the audit-replay path has no equivalent, so a same-version run larger than MAX_PAYLOAD accumulates whole, checkExcessMessageSize throws at the next version change (~5060, the deliberate no-data-loss throw), the leg closes, resumes from the un-advanced cursor, and re-encodes the same run forever. Every duplicate shares one localTime key, so no cursor value can express progress inside the run — a size-based flush needs sub-key resume semantics, which is presumably why the flush-on-version-change design exists. Note this also wedges on any legitimately huge single transaction.

A third contributing behavior: the duplicates are appended at localTime = version, i.e. behind newer keys already in the log, violating the txnlog's monotonic-timestamp assumption ("fixed size index entries for fast traversal using binary search"). Resume cursors newer than the run's key do not skip it, because iteration past the binary-searched start position is append-ordered.

Production evidence ([customer-cluster], 2026-09-01)

• Seed: the bridge peer row hdb_nodes/[bridge-peer] is caught in a repeating authenticated remove_node/re-add cycle on the bridge node (tracked separately). One such delete committed at version 1788239374542.4155 (05:09:34.542Z), present exactly once in the origin node's local log.
• 06:08:00Z: mesh-wide reconnect wave (all 15 peers × data/system/redirects legs re-established, some twice).
• 06:12:19–06:28:15Z: every non-bridge node's eva-us-sea-1 (origin) transaction log grew from ~16 MB to 725–800 MB — e.g. us-west-2: 50 × 16 MiB .txnlog files. Parsing them offline: ~6.4M entries that are byte-identical delete entries for that row, all with version 1788239374542.4155, all keyed at that same timestamp, user_name = the bridge's replication credential. The origin node holds the delete once and has zero duplicates (74 MB local log) — receivers minted every copy.
• 06:22:14Z onward: Message too large to send, size: <n> ... database: system from 15 of 16 nodes; sizes grow 470,492,640 → 678,678,033 while echo inflow continues, then freeze (us-west-2: 678,695,309 ≈ 6.4M entries × ~105 wire bytes). 39–481 errors/hour/node since (warn-throttled).
• Legs pinned: lastReceivedRemoteTime shows 05:38:31Z on 14 of 15 peers per node (the max-version watermark from the last successfully applied writes — two hdb_nodes patches). Only the origin's direct legs still deliver (its log has no run).
• The 100 MB cap is what stopped the echo: once a node's run exceeded MAX_PAYLOAD its fan-out of the run wedged, and by ~06:34 all fan-out legs were wedged. The wedge is the storm's only circuit breaker.
• Precedent: identical error signature on 2026-08-17 (~394 MB), gone after retention purged the entries. The current run ages out of the 3-day window on ~2026-09-04.

Reproduction sketch

1. 3+ node v5.2.7 mesh (RocksDB), any replicated database.
2. Write and replicate a row; then delete it, so a delete audit entry (version V) exists in every node's origin log for node A within the retained window.
3. Force reconnect churn such that the delete is re-delivered outside the leading-duplicate tie window (e.g. bounce legs, or re-deliver via an indirect leg whose per-origin startTime predates V — indirect legs rarely advance per-origin cursors for origins they don't primarily serve).
4. Observe: each receiver re-applies the no-op delete and appends a fresh audit entry at key V; peers pick each other's copies up and re-log again; copies multiply per exchange round.
5. Once a node's same-version run exceeds replication_maxPayload, every leg carrying it throws Message too large to send at sendQueuedData (called from the version-change flush in sendAuditRecord), closes, resumes from the un-advanced cursor, and repeats indefinitely. The database's mesh replication is permanently wedged while inflow from not-yet-wedged legs keeps growing the run.

Impact

• Full system-db (or any-db) mesh replication outage: node/user/role/schema state stops propagating cluster-wide.
• The retry loop re-encodes hundreds of MB per attempt per leg ("JavaScript execution has taken too long" watchdogs fire).
• Self-heals only by auditRetention file purge, days later, and only until the next delete is re-delivered.
• Raising replication_maxPayload un-chokes the echo and makes it worse (see below), so there is no safe config workaround.

Suggested fixes

1. Dedup replicated deletes (primary). On applying a replicated delete with no stored record, consult the audit log for an existing entry at (version, tableId, id, nodeId) — the same keyed lookup the CRDT path already uses at Table.ts ~2667 — before appending a new audit entry; skip apply+relog on a hit. Equivalently: extend isDurableIdentityTie/the leading-dup window with a per-origin version floor so version <= floor deletes of absent rows are skippable. Deletes are currently the only entry class with zero dedup layers.
2. Bound the audit-path frame (the Replication silently and permanently drops any transaction whose frame exceeds replication_maxPayload #711 analog). A size-based flush inside a same-version run requires resumable sub-positions (e.g. counting entries within a localTime key in the sequence update, or splitting the run into multiple frames the receiver treats as one transaction). Absent that, at minimum detect the condition (frame > cap with unchanged txnTime) and surface a distinct, actionable error naming the run's version/table/id instead of retrying the identical encode forever.
3. Reject out-of-order relog keys. Appending at localTime = version behind newer keys breaks the log's binary-search contract and pins resume cursors; relogged entries could be keyed at arrival time (preserving version inside the entry) or the duplicate append avoided entirely per fix 1.

Fix 1 alone prevents the storm; fix 2 alone prevents the permanent wedge; both are worth having.

Lavinia, via Claude