Stop replicated deletes from echoing across the mesh, with an offline repair for logs already affected

[kriszyp]

Stops a mesh from echoing a re-delivered replicated delete into millions of audit entries, and adds an offline, operator-run tool that compacts runs older releases already wrote into RocksDB transaction logs, so a sender wedged on replication_maxPayload can ship again. The root cause is fixed in core by the merged Stop re-logging a replicated delete that is re-delivered onto its own tombstone, pinned here at its core-main commit. The repair is repairDeleteEchoRuns.ts with its CLI bin/repairDeleteEchoRuns.ts. No Pro runtime code changes: the earlier sender-side collapse is removed, and replicationConnection.ts is identical to main.

Closes #826

Depends-on: HarperFast/harper#2761

For the human reviewer

1. Requirement. Per the thread, this PR swaps the permanent sender-side collapse (a per-entry branch and buffer copy in every replication stream, adjacent copies only) for an isolated tool that is run once where needed. With the core fix on every node, new runs stop, and the tool exists only for logs written before then. The alternatives cost more:

   • waiting for the 3-day retention purge stalls the leg, then skips the run's key, so the peer never gets those deletes;
   • purging logs and re-cloning drops all undelivered history in the purged files.

   Temporarily raising replication.maxPayload on both ends works for a modest run (a reviewer suggestion). It does not scale to a million-copy run, because the whole frame is built in memory.

2. Size. The tool is about 1,140 lines plus 725 lines of unit tests. Most of it is safety, not the dedup rule (about 60 lines at the drop decision):

   • a crash-safe publish order and a manifest;
   • an exact --restore;
   • an exclusive database hold;
   • checks at every step (ownership and symlinks included).

   A much smaller tool would rewrite in place and leave backups to the operator (cp -a transaction_logs); that trades crash safety and restore for about 300 lines.

3. Exclusion is RocksDB's own lock, held for the whole run. withDatabaseHeld opens the database writable, which takes the kernel fcntl lock on <db>/LOCK, so Harper cannot open the database until the repair ends, from any process or pid namespace. The handle's transaction logs point at an empty directory: a store over the real logs purges old files on close, and every handle in the process shares it. That is also why the read-back runs in a child process. Earlier rounds used /proc/locks (Linux 4.9+ filters it by pid namespace, so a sibling container is invisible) and hdb.pid (wrong across namespaces). Cost: the writable open rewrites RocksDB's MANIFEST, CURRENT and OPTIONS and runs WAL recovery, exactly as a Harper start does; the logs are untouched (probed).

4. What counts as a copy. A delete is dropped only when both hold:

   • its replicated bytes, everything after Harper's local prelude, equal those of a delete of the same record kept earlier in the same one-timestamp run;
   • no other entry for that record lies in between.

   [delete K, put K, delete K] in one transaction is therefore kept whole. A run continues into the next file when the previous file closes its last transaction, which is how the sender frames it (scanOptions). After a transaction split across files by an older writer, dedup restarts, so the transaction keeps an entry in each file to carry its last flag. A run whose comparison state exceeds 256 MiB refuses its file rather than being partly repaired.

5. Positions other readers depend on:

   • txn.state is rewound to the file header while that file is swapped, then remapped (publishStore). A crash between the two replays already-flushed writes; it never skips unflushed ones.
   • A rewritten newest file gets a header-only successor stamped with the store's latest timestamp. rocksdb-js starts a query in the newest file whose header timestamp is below the start. The integration test caught a copied, older stamp skipping file 1 and losing B's z delete: 2/2 failed without the fix, 5/5 passed with it. The new sequence also means no stale derived-index coverage position can equal the tail.
   • Files with a non-zero append boundary, a torn tail, an unclosed newest transaction, or an unknown side file are refused, and the CLI exits 1.

6. Format coupling. decodeEntry and the file walker re-implement two layouts: rocksdb-js's v1 log layout, and Harper's prelude and audit-entry field order. The unit tests write their entries through core's own RocksTransactionLogStore.put, and the tool refuses unknown format versions and side files. Moving the decoder into rocksdb-js or core would mean a release there for a one-off tool; a layout change needs a tool change.

7. Operator contract. The tool only discovers <root>/database/*; custom storage.path layouts are not supported. Started as root, the CLI becomes the databases' common owner, and the library never re-owns files. Originals stay hard-linked in <db>/transaction_logs.repair-* until someone deletes them, so a clone or backup of the database dir carries them along. Rollout:

   1. upgrade every node;
   2. on each wedged sender, stop Harper and its supervisor;
   3. dry run, --apply, restart;
   4. verify record-level convergence;
   5. delete the backup.

8. Declined.

   • The core allocation finding (a {txnLogKey, nodeId} object per re-delivered delete of an absent record) is in the merged harper#2761; it's a core follow-up.
   • Not built: SIGKILL child-process crash tests (crash points are exercised by injected throws after each durable step), ENOSPC, a real HNSW coverage test, and fixtures from every historical log writer.
   • test:integration:all did not complete locally (see Verification); CI runs it.

Changes

• core — submodule bump to df188d961, the merged harper#2761, a direct descendant of main's pin.
• replication/repairDeleteEchoRuns.ts:
  • the streaming file compactor, with last-flag relocation at the drop branch;
  • inventory, which refuses unknown files, links and extra hard links;
  • planning, with strict native validation;
  • staging, under a preparing manifest, auto-removed only when it's the tool's own (cleanup);
  • publish, then byte-level verification, including the replaced inode itself, then the read-back;
  • restore, which checks every source, then copies through one O_NOFOLLOW descriptor, after a validated manifest;
  • per-database reports, where one failure never hides the others.
• bin/repairDeleteEchoRuns.ts — dry run by default, --apply, --restore <backup-dir>; exit 1 when any file is refused.
• replication/DESIGN.md item 20 records the invariants and the key-files row; the sender-side entry it replaces is gone.

Verification

• Unit: unitTests/replication/repairDeleteEchoRuns.test.mjs, 40 passing. Real logs are written through rocksdb-js and core's RocksTransactionLogStore.put. It covers:
  • single-record, interleaved and cross-file runs, [delete, put, delete] kept whole, last-flag relocation;
  • txn.state remap plus startFromLastFlushed replay;
  • the tail file, including a query from a cursor between the header and the run, which fails without the stamp fix;
  • older writers' split transactions;
  • refusals: torn tail, boundary marker, off-boundary flush, unclosed newest file, unknown file, hard link, memory budget, oversized entry, symlinked or dangling database, another process holding the database (a child process);
  • an injected crash after each of the 6 durable apply steps, then strict validation, read-back, replay decodes, byte-exact --restore, and a successful retry;
  • interrupted-staging cleanup vs. an unrecorded directory; mid-repair write detection; a crafted manifest with path traversal; symlinked or doubly-linked restore sources;
  • an untouched file older than retention surviving;
  • CLI exit codes.
• Integration: integrationTests/cluster/deleteEchoRun.test.mjs, with the fixture writing raw clones of the real delete:
  • Setup: two nodes, 256 KiB cap. B is offline while A gains a z × 20,000 run.
  • Repair: A is stopped, and the built CLI reports and then drops 20,000 copies. A restarts; its log holds one z delete and z is still deleted.
  • Receiver check: [x, y] × 40 is then planted below the cap, and B must log one delete per record.
  • Results: passes on the final head. With --apply removed, it fails: the marker never reaches B within 90 s, and the sender logged too large to send 38 times.
  • config.yaml and schema.graphql declare the fixture's REST table.
• Gates: npm run test:unit 1284 passing / 1 pending. npm run typecheck clean. oxlint and prettier clean on every changed file.
• Not run locally: test:integration:all in 4 local shards did not complete. The shards were starved of loopback addresses by other Harper nodes on a shared box (load average about 24), so I stopped them; CI runs the suite. Pro runtime code is identical to main; the only runtime change is the core pin, one commit, which core CI covered in harper#2761.
• Review: planning review cleared twice (Framing-Verdict: chosen-approach-sound, including the round-5 recheck). Implementation review of this change ran 2 full and 5 delta rounds (codex, gemini, cursor-muse, plus Harper domain adjudication), until it converged on a nit-only round. Every finding was fixed or ruled on in item 8 above. One decision was overruled and later adopted: holding RocksDB's lock for the whole run, once a writable open with a private logs path proved to leave the logs untouched.

Complexity: complicated

🤖 Generated with Claude Code — Claude Opus 5.5

Origin — the dispatch brief this PR was written from

Stop replicated deletes from echoing across the mesh, with an offline repair for logs already affected

Maintain #895 (Stop replicated deletes from echoing across the mesh, with an offline repair for logs already affected) on branch fix/replicated-delete-relog-storm. Read the current PR head, mergeability, and latest check runs before acting; the dispatch observation may be stale. First read the remote head for refs/heads/fix/replicated-delete-relog-storm without updating local refs and require it to equal this task's observed head 3ada4ca; if it differs, stop with needs-input because remote history changed after dispatch. Then fetch only refs/heads/main from origin into refs/remotes/origin/main. If Git reports a paused rebase for this task's exact generation branch, require that rebase's recorded original head to equal 3ada4ca and its recorded onto commit to equal the fetched origin/main tip; stop with needs-input on either mismatch. Only then resume it without repeating the local-HEAD ancestry check or starting another rebase. Otherwise record the remote SHA, verify it is an ancestor of local HEAD, and rebase onto the fetched base. If Context carries companion instructions, follow them exactly for every named gitlink; they override ordinary conflict handling. Preserve both sides' intent for every other conflict. Stop with needs-input on semantic conflicts. Force-with-lease is authorized only for this rebase. Immediately before pushing, re-read the PR's base ref and body plus every companion PR head/state named in Context. The base must still be main, and the declarations and submodule bindings must still match Context. If an open companion advanced, update its named gitlink to the new exact head and rerun relevant tests. If it merged, point the named gitlink at the companion repository's current default-branch tip after verifying that tip contains the merge, then rerun relevant tests. If it closed without merging, stop with needs-input and do not publish its abandoned head. If the base, declaration, binding or any other companion state changed ambiguously, stop with needs-input. Then push the rebased head with git push --force-with-lease=refs/heads/fix/replicated-delete-relog-storm:EXPECTED_HEAD_SHA origin HEAD:refs/heads/fix/replicated-delete-relog-storm, substituting the recorded SHA. If the remote head changes, stop with needs-input; never overwrite intervening remote work. Do this before evaluating CI. Then inspect CI for the resulting current PR head, not old-head failures. Wait for relevant pending checks, diagnose remaining failures, fix them, run relevant tests, and push in THIS task. Do not create a separate CI-fix task. Stop with needs-input for judgment calls or an unavailable CI result; report exactly what was verified. No merge is authorized.

COMPANION PRs — this PR declares Depends-on: and its diff moves a submodule pointer, so that pointer is NOT an ordinary conflict: it has no "both sides" to preserve. Take neither side of the gitlink conflict; set each pointer named below to exactly the sha named with it, and say in # Log that you did. Immediately before push, re-read the PR body and each companion's state/head as the task requires; these SHAs are a dispatch-time snapshot, not a lease on another repository.

• core -> Stop re-logging a replicated delete that is re-delivered onto its own tombstone harper#2761 is MERGED. Its commits are on that repo's default branch now, so point core at the default branch rather than at the companion's branch commit.

_{Dispatch: task pr-maint-eb7cb0115f26bdbce455d91d36ab1ce7 · queued by automation · ran by claude/sonnet/xhigh · worker kzyp-xps-1}

_{Review-Coverage: authored=claude; ran=cursor-muse,gemini,codex; adjudicated=domain; declined=cursor-grok,cursor-composer,cursor-kimi; rounds=12; full=3 @ 12a5171}

_{Human-Review-Need: 4 (decisions: offline-byte-rewrite, format-knowledge-in-pro, writable-open-as-lock, coverage-invalidation-via-tail-file, no-cluster-version-gate) @ 12a5171}

[gemini-code-assist]

Code Review

This pull request addresses an issue where echoed delete runs could wedge the replication sender or cause excessive re-logging on the receiver (harper-pro#826). It introduces a mechanism to detect and drop adjacent byte-identical deletes within the same frame using a new repeatsQueuedDelete helper. The changes are accompanied by comprehensive integration and unit tests, as well as updated design documentation. The review feedback suggests a minor improvement in replicationConnection.ts to use the standard and more idiomatic TypedArray.prototype.set method instead of Buffer.prototype.copy.call for copying bytes.

[claude]

Reviewed; no blockers found.

[claude]

Suggestion (non-blocking): This spawnSync has no timeout, unlike the integration test's own CLI-invocation helper (runRepairTool), which passes timeout: 120000. If the read-back child hangs — e.g. the rocksdb-js iterator stalls on an unusual database — withDatabaseHeld's action() never returns, so the writable RocksDB handle (and its LOCK) stay held until an operator notices and kills the process by hand. A timeout here would turn that silent hang into the same clear, bounded failure this tool gives for every other verification step.

Suggested change

	{ encoding: 'utf8' }
	{ encoding: 'utf8', timeout: 120_000 }