Replication silently and permanently drops any transaction whose frame exceeds replication_maxPayload #711

Summary

An outgoing replication frame larger than replication_maxPayload (default 100 MB) is logged and then silently discarded, with no throw, retry, chunking, or backpressure. The client that wrote the transaction gets a success response, the peer never receives the records, and the loss is permanent. Confirmed on a live two-node v5.2.2 cluster.

Where

replication/replicationConnection.ts, in sendQueuedData(): if (checkExcessMessageSize(frame.position - frame.encodingStart)) return;. On an oversized frame checkExcessMessageSize logs Message too large to send, size: N and returns true, so the function returns before ws.send(). The identical predicate on the blob-chunk path throws (throw new Error('Blob chunk too large')), but the audit-frame path swallows it. The websocket server is configured with maxPayload: 10 * 1024 * 1024 * 1024 (replication/replicator.ts), 100x the sender's own guard, so the transport is not the constraint.

Reproduction (live, v5.2.2)

Two-node cluster, insert one transaction whose encoded replication frame exceeds the limit (a bulk insert, or a handful of large records):

• At the real 100 MB default: one insert of 1200 records x 100 KB produced a 120,081,888-byte frame. Sender logged Message too large to send, size: 120081888 remote node: nodeB database: data. Client insert returned 200 {"message":"inserted 1200 of 1200 records"}. The peer received 0 of 1200 (queried directly, not via logs). Same behavior at a lowered replication_maxPayload=1000000 with a 2 MB frame.
• Not a wedged leg: subsequent smaller transactions replicate normally, the socket stays connected: true, backPressurePercent: 0.
• Permanent: sendAuditRecord advances sentSequenceId = localTime before sendQueuedData(), and the end_txn sequence trailer of the next transaction advances the receiver's resume cursor past the dropped one. On receiver restart the resume cursor was already at the next transaction's timestamp, so the dropped records are never re-requested.

Impact

Silent, permanent data loss on replicas for any transaction over the payload limit (bulk loads, large-record writes). The only recovery is a full re-copy (add_node isLeader:true / startTime=0), which normal operation never triggers.

Fix

Draft PR converts the silent drop into a loud, recoverable failure (throw so the leg closes and resumes from the un-advanced cursor rather than skipping). The proper long-term fix is mid-transaction chunking (split a large transaction into sub-maxPayload frames), which is a larger protocol change tracked as follow-up.

Lavinia, via Claude