Replication: transitive/proxied re-delivery floods peers with already-applied out-of-order writes (reduce volume; complements harper#1310) #399

Summary

Transitive/proxied replication re-delivers already-applied writes to peers out of order, generating a high volume of redundant out-of-order applies. On records with deep audit histories this drove the core out-of-order resequencing walk to its depth cap repeatedly, starving the event loop (→ replication ping-timeouts, wedged subscriptions). Observed in production on [customer-cluster].

This is the "reduce the volume" counterpart to harper#1310 (which absorbs these cheaply in core via an up-front keyed dedup). #1310 stops them from being catastrophic, but they're still generated and still cost work every cycle — this issue tracks stopping them at the source.

Evidence ([customer-cluster], [node] on 5.1.1)

A non-pausing logpoint at the core depth-cap site captured the capping writes:

• type=patch, fullUpdate=false, every one carrying viaNodeId (relayed via a proxy node, not direct).
• Multiple source nodes (srcNode 4/8/9), all older than the local record head (out-of-order).
• The post-cap keyed lookup matched (dupVerMatch=true, dupNode===srcNode) — i.e. they were exact (version, nodeId) duplicates already applied, redundantly re-delivered via the relay path, each forcing an ~858–1000-step walk before being discarded.

So the re-deliveries are real, redundant, and proxied.

Suspected mechanism

The proxy-resume start-time derivation in replication/replicationConnection.ts (~the indirect-connection block that reads a proxy node's seqId, around the Using sequence id from proxy node path) appears to re-stream a longer already-applied tail than necessary — when the proxied seq cursor isn't relayed/persisted as tightly as the direct cursor, the leader streams from too-early a point and re-delivers writes the follower already has. (High confidence on the symptom — proxied out-of-order duplicate re-delivery — from the logpoint; the exact start-time-relay gap as the generator is the leading hypothesis from reading the resume code, not yet confirmed end-to-end.)

Relationship to existing work

• harper#1310 — up-front keyed dedup in Table.commit absorbs these in O(1) instead of an O(depth) walk. Mitigates impact; does not reduce volume.
• perf(replication): fast-skip leading duplicates on resume #370 (leading-duplicate fast-skip) is deliberately un-armed for proxied/indirect subscriptions (hasPersistedResumeCursor checks the direct cursor only). Extending its arming to proxied subscriptions would give an earlier, cheaper skip for the in-order proxied subset.
• replayLogs persists replicated structure-updates to a key the RocksDB decode path never reads (latent shared-structure decode-loss) #362 / harper#1275 — the shared-structure decode root cause (separate; already fixed).

Proposed directions

1. Tighten the proxy seq-cursor relay so a proxied resume starts from an accurate point and doesn't re-stream an already-applied tail (attacks the cause).
2. Arm perf(replication): fast-skip leading duplicates on resume #370's leading-dup-skip for proxied subscriptions (per-origin-node cursor from the proxy seqId) — earlier skip for the in-order subset.
3. Consider whether redundant transitive delivery is needed at all when a follower already receives the writes directly (reduce proxied fan-in).

_{Filed from the [customer-cluster] incident investigation. Investigation + writeup by Claude (Opus 4.8).}