copyApply: snapshot non-system base-copy rows without audit entries (part of #480)

[kriszyp]

Summary

Receive-side half of copyApply (paired with harper#1479). Base-copy frames are applied as current-state snapshots — no audit/transaction-log entry, no out-of-order resequencing — instead of being re-fabricated as transactions. Removes the O(n) keyed audit-dedup spin and the per-peer transaction-log amplification when copying an analytics-laden table (#480).

Purpose

Copying a table whose rows predate transaction-log retention makes every applied row's keyed dedup miss and scan to end-of-log (one worker ~100% CPU, no forward progress), and re-fabricating copied rows as transactions floods the per-peer logs that other peers then copy. See #480.

Design — please focus review here

• C2 gate (event.isCopyApply = messageIsCopyFrame && auditRecord.version < copyModeStartTime): only rows older than copyStartTime are snapshotted. The post-copy audit replay resumes from copyStartTime and re-delivers every row >= copyStartTime, so those keep real audit entries and the redelivery dedups (a commutative patch would otherwise double-apply). Older rows are never re-delivered → safe to snapshot. RocksDB-only (core gates on isRocksDB).
• Latched copy-frame status (messageIsCopyFrame, once per WS body): the decode loop awaits (waitForDrain/setImmediate) and ws does not serialize async handlers, so a later COPY_COMPLETE could flip copyCompleteReceived mid-body and drop trailing rows to the audited path.
• Durability (the load-bearing part): copied DBs run WAL-off and these rows carry no transaction-log entry, so the persisted resume cursor and the [seq] watermark must only advance behind an explicit RocksDB flush (memtable → SST):
  • flushDurableCopyCursor flushes on a replication_copyCursorFlush{Bytes,IntervalMs} cadence (64 MB / 5 s) and persists the copyCursor only after the flush resolves; maybeFinishCopy forces a final flush before removing the cursor.
  • seqUpdateEndTxn wraps every plain sequence-update end_txn (SEQUENCE_ID_UPDATE, REMOTE_SEQUENCE_UPDATE, blob-drain re-emit); its onCommit flushes the snapshot rows before [seq]=copyStartTime is persisted. core awaits onCommit then calls updateRecordedSequenceId, so this orders flush-before-seq; a flush rejection skips the seq persist so the copy re-runs rather than losing rows. The per-batch endTxnEvent.onCommit carries the same gate (defense-in-depth).
  • Crash before a flush → re-copy idempotently from the last durable cursor (put-if-newer). Dual-engine flush (RocksDB flush() / LMDB flushed).

Relationship to #483

Complementary and non-overlapping (#483 paces the send-side socket flush; this is the receive-side apply). Shares the 5 s cadence + replication_copy* env convention; can stack/rebase once #483 lands. (Did not reuse #483's createCopyFlushPacer to keep this PR independently mergeable; worth consolidating after #483 merges.)

Review notes

Cross-model reviewed (Codex, 4 rounds): CRDT redelivery double-apply → C2 gate; seq cursor not gated on flush → seqUpdateEndTxn at all emit sites; LMDB flush()/replay-cursor divergence → RocksDB-gating + dual-engine flush; async COPY_COMPLETE interleaving → latched status. No open findings; a Gemini pass is running and I'll fold anything it surfaces. Coverage analysis: C2 + the interim retention-guard (separate) cover the mass O(n)-spin paths, so the txn-log index work is optional (demoted).

Generated by an LLM (Claude Opus 4.8).

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[claude]

Reviewed; no blockers found. The new commit (fcc403d) is a mechanical core submodule pointer bump. Findings 1 and 2 from the prior round (copyFlushRetryTimer cleanup on WS close; copyFlushInFlight not reset on COPY_START) remain in the code — re-evaluated as non-blocking on re-analysis, but worth a follow-up cleanup.

[kriszyp]

Scope clarification (post-CI iteration).

copyApply applies to non-system databases only. The system DB drives cluster machinery off the audit aftercommit stream — hdb_nodes → subscribeToNodeUpdates (peer discovery / connection setup) and hdb_certificate → CA install — which copyApply's no-audit writes suppress, so a freshly-copied node couldn't form its cluster (the integration-test failures on shards 1/2/4 were exactly this). It's now gated via copyApplyActive() = non-system RocksDB; the system copy reverts exactly to main's behavior.

Consequently this PR does not fix #480's system-analytics copy spin (hdb_analytics lives in the system DB) — that is handled separately by the retention-horizon dedup guard, which removes the O(n) keyed-lookup cost without dropping audit entries. This PR is the non-system base-copy optimization (perf + no txn-log amplification on data copies).

Also added: backoff (not busy-loop) on a persistent copy-cursor flush failure.

Possible follow-up (discussion): a txn-log "table-reload" marker would let copyApply safely cover the system DB too — subscribers that need back-filled data react to the marker by reloading from the table, instead of relying on per-row events.

— generated by an LLM (Claude Opus 4.8)