Replication W10: Performance #439

Workstream W10 of #430 · performance

Summary

Hot-path and structural throughput improvements identified from the protocol read. The incremental items stand alone; the largest wins ride the foundational workstreams (per-origin parallel apply from W4, dedicated threads from W6).

Pieces

• Direct binary relay for the copy path. Today the base copy decodes every record only to re-encode it (TODO: eventually we don't want to decode, we want to use fast binary transfer). A passthrough relay would cut copy CPU substantially for large tables and shrink the system base-copy gated on huge hdb_analytics (analytics.replicate:true) blocks hdb_deployment convergence → deploys fail after any copy #421/NATS MAX_PAYLOAD_EXCEEDED error when cluster cloning large data #266 surface.
• Audit-entry field elision in multi-entry transactions (the TODO to elide repeated version/username across entries in one txn) — bandwidth win.
• Avoid the per-skipped-record setImmediate promise allocation in the inbound decode loop (frequent on exclusion-filtered tables).
• Make MAX_OUTSTANDING_COMMITS, RECEIVE_EVENT_HIGH_WATER_MARK, BACK_PRESSURE_INTERVAL runtime-configurable (currently hardcoded closure constants) so tuning doesn't require a code change.
• Bound the leading-dup fast-skip store read on the hot path (it does a getEntry/fsPromises.access per candidate from known-lagging nodes).
• cluster_status getSystemInfo cost (cluster_status: avoid full getSystemInfo call for performance #260) — shared with W8.
• Structural (rides W4/W6): per-origin parallel apply; dedicated replication threads.
• Validation: WAN latency + high-frequency small-record pattern, 500K records (Stress: extend replicationLoad to 500K records + WAN latency + high-freq small-record replication pattern #300).

Retires / advances

• Stress: extend replicationLoad to 500K records + WAN latency + high-freq small-record replication pattern #300 — extend replicationLoad to 500K + WAN + high-freq small-record
• system base-copy gated on huge hdb_analytics (analytics.replicate:true) blocks hdb_deployment convergence → deploys fail after any copy #421 — large hdb_analytics base-copy blocks deploy convergence (copy-path + prioritization angle)
• NATS MAX_PAYLOAD_EXCEEDED error when cluster cloning large data #266 — large-clone payload (copy-path angle; transport chunking is in W3)
• cluster_status: avoid full getSystemInfo call for performance #260 — cluster_status getSystemInfo cost

Dependencies

Incremental items are independent. Structural wins depend on W4 and W6.

Effort / risk

M / low–medium (incremental); structural wins are sized under W4/W6.

Acceptance criteria

• Large-table base-copy throughput improves measurably (binary relay).
• The three runtime tunables are adjustable without a rebuild.
• The Stress: extend replicationLoad to 500K records + WAN latency + high-freq small-record replication pattern #300 stress pattern passes within target latency.

🤖 Filed by Claude on behalf of Kris.