Stop re-logging a replicated delete that is re-delivered onto its own tombstone

[kriszyp]

A replicated delete that reaches a node already holding the tombstone it left no longer re-applies and appends a fresh audit entry. In a mesh each re-logged copy was new log tail that every peer forwarded and re-logged in turn, so one re-delivered hdb_nodes delete echoed into ~6.4M identical entries per node and wedged system-database replication (Mesh replication amplifies a re-delivered hdb_nodes delete into millions of duplicate audit entries).

Refs HarperFast/harper-pro#826 — the harper-pro PR that bumps core to this branch carries the closing keyword and the sender-side half.

For the human reviewer

1. Requirement. The issue asks for three fixes; this PR is fix 1 (dedup replicated deletes), which is the root cause: deletes were the only write type with no identity-tie check. Fix 3 (duplicates appended behind newer keys) disappears with it — a first-time application keyed at the origin's log key is the dual-clock design, not a defect. Fix 2 (a same-key run above replication_maxPayload) is handled for logs older releases already wrote by the companion harper-pro PR; a single legitimate transaction above the cap is a pre-existing limit left alone. As specified, yes.
2. Layer: the delete commit, not the transport. Chosen: the delete handler recognizes its own re-delivery. Alternative: the replication transport stops re-forwarding entries it already logged under the origin's key. The transport route would also cover composite transactions (item 4); the handler no-op can stay as a backstop either way, so switching later is cheap. A receive-path skip in harper-pro was rejected on the facts: it reads committed state before the frame's transaction, so N first-seen copies in one frame all flow (measured 51 entries), and it is an optional optimization (replication_leadingDuplicateSkip).
3. Identity is origin node + log key, via isAuditEntryWrite — never the record version (the dual-clock rule in resources/DESIGN.md). A distinct delete that reuses the version under a new log key still applies. The no-op still stages the removal, so every later write in the transaction sees exactly what it saw before; state after a skip equals state after today's re-apply.
4. Composite transactions are not fully covered. A delete that follows another write to the same key in its transaction still applies — it is not a tie with its own removal. So a re-delivered [delete K, put K], or a [put K, delete K] whose put re-applies inside the harper#1137 keyed-lookup lag window, still re-logs, as today. Closing that needs transaction-level "already applied" knowledge (item 2's transport route). [put K, delete K] with a readable log is pinned at one entry each.
5. The chain walk leans on write.skipped. removalBefore walks back past writes marked skipped and stops at any other unstaged write (an invalidate or relocate stores a null stub without staging it; an INVALIDATED stored entry never counts). A future handler that sets skipped after storing something would let a delete be taken as already applied. Alternative: an explicit "published no state" marker on invalidate/relocate/publish writes.
6. Known gap, accepted. A null stored by a publish or a residency-local relocate carries no marker. If one origin transaction ever arrived split with the delete alone in a later frame, that delete would log nothing (the key is null either way; subscribers and relays would miss the delete event). Ordinary replication keeps one log key in one frame.
7. LMDB keys its log by local time, so a stored tombstone there has no origin log key: cross-frame re-deliveries still apply (fail open, as today); copies within one transaction collapse. Replication runs on RocksDB.
8. Declined review findings: an extra { txnLogKey, nodeId } literal on absent-row deletes only (small next to the audit write it avoids); two local deletes of an absent key colliding on one timestamp merge (same log key, same state, nothing lost).

Changes

• resources/Table.ts — the delete commit returns without writing when the key's no-record state was produced by this same write, still staging the removal; every applied delete stages { value, localTime, nodeId } so a later copy in the transaction can tie against it; removalBefore finds that state; the handler now resets and sets write.skipped like the other commit-handler early returns. Look hardest at removalBefore: its stopping rule is what keeps an invalidate/relocate stub from passing for a tombstone.
• resources/auditStore.ts — isAuditEntryWrite takes only the two identity fields it reads (type-only).
• resources/DESIGN.md — the invariant, where it is enforced, and its limits; DESIGN.md — its index line.

Verification

• unitTests/resources/replicatedDeleteRedelivery.test.js — 16 cases through a sourceApply transaction at the origin log key, one of them through the real sourcedFrom replication apply dispatcher. RocksDB 16 passing, LMDB 9 passing / 7 pending (engine-specific). On base 5 fail: 3 re-deliveries → 4 entries, one frame carrying the delete 50× → 51 entries, dispatcher 21 copies → 2. The two [invalidate, delete] stub cases and the [put, delete] count case each fail on the intermediate guard that review rounds 1 and 2 caught.
• npm run test:unit:resources:core 2534 passing / 43 pending; test:unit:resources:indexes 459 passing; LMDB test:unit:resources:core 2062 passing, :indexes 244 passing. test:unit:main 5771 passing, 32 failing — all environmental in this submodule checkout (no core/node_modules: shrinkwrap-pin canaries and global-isolation tests look for installed packages there; one file-watcher test), none touching resources/.
• End to end through a real two-node replication hop: harper-pro integrationTests/cluster/deleteEchoRun.test.mjs on the companion PR. With this core change reverted, the receiver logs 41 deletes for a record whose echoed run it receives in one frame; with it, 1.
• npm run check:design-docs, prettier, oxlint clean on the changed files.

Complexity: complicated

🤖 Generated with Claude Code — Claude Opus 5.5

_{Review-Coverage: authored=claude; ran=gemini,cursor-muse,codex; adjudicated=domain; declined=cursor-grok,cursor-composer,cursor-kimi; rounds=3; full=1 @ 93309f7}

_{Human-Review-Need: 3 (decisions: receive-side-idempotence, skipped-marker-walk, stored-entry-type-blind, lmdb-partial) @ 93309f7}

[gemini-code-assist]

Code Review

This pull request implements a mechanism to ensure that a replicated delete re-delivered onto its own removal is treated as a no-op in Table._writeDelete, addressing harper-pro#826. When a key has no record because the same write (identified by origin node and log key) removed it, the delete will not write anything (avoiding redundant index changes, tombstone rewrites, or audit entries) while still staging the removal. This prevents infinite log tail growth in mesh environments. The changes include updates to DESIGN.md, logic in Table.ts (with a new removalBefore helper), a signature adjustment in auditStore.ts, and a comprehensive suite of unit tests in replicatedDeleteRedelivery.test.js. There are no review comments, and I have no feedback to provide.

[claude]

Reviewed; no blockers in this diff.

Follow-up (non-blocking, outside this diff)

The PR states "deletes were the only write type with no identity-tie check," but _writeInvalidate (resources/Table.ts:2664) and _writeRelocate (resources/Table.ts:2724) appear to share the pre-fix defect: both call updateRecord unconditionally once precedesExistingVersion(...) < 0 fails to bail (a redelivery ties at 0, not <0), with none of _writeUpdate's dedup or this PR's new removalBefore/isAuditEntryWrite check. invalidate is in transactionBroadcast.ts's real-time ACTIONS_OF_INTEREST fan-out, so a redelivered invalidate (reachable from the same replication switch as delete) would re-append and re-forward the same way delete did pre-fix; relocate is excluded from that live path but still replayed via the startTime backfill loop. Worth a tracking issue analogous to this one, scoped to those two handlers.

[maurice-harper]

Production evidence for this shape from a 2026-09-24 incident (10-node cluster on harper-pro 5.1.12; reproduced the same day on the customer's 5.1.28 staging cluster), collected by a read-only decode of the system transaction logs on two nodes. X1/X2 = nodes removed from the cluster three months earlier via remove_node, whose per-origin logs were still present.

• Node A, X1 log: 7,054,244 DELETEs of hdb_user clone-temp-admin at key 2026-09-16T14:32:32.702Z and 2,083,541 at key 2026-09-24T02:16:57.766Z, every one with previousVersion == version, origin id = X1.
• Node B, X1 log: 9,469,137 at the first key; node B's logs for two live peers: 732,673 and 2,234,304 at the second key.
• X2 log: 16,572 DELETEs of two hdb_nodes rows, same shape.

That is millions of copies of one delete at one version, which is what filled the stores (16 MB segments every 1 to 2 min per node) and the worker heaps. This fix is merged to main only; the 5.1 patch the customer is waiting on should carry it, since the per-origin-log fix in #2793 stops the loop but not the flood.

Full frame breakdown with origin ids is on #2793.