Summary
A bulk-copy resume trusts the follower's persisted cursor: everything in copy order before {currentTable, afterKey} is assumed delivered, and the sender skips straight past it. Nothing ever verifies that assumption against actual table content. When a cursor outruns delivery (the harper-pro#537 poisoned-cursor class, or any future variant), every resume re-seals the hole: the skipped range is never re-examined, the counts sit thousands of rows apart, and no signal fires anywhere.
The reworked stream-count verification (harper-pro#538) covers in-flight loss within one copy session. It cannot see at-rest holes from earlier sessions, which is exactly where poisoned cursors live. In the #538 review we converged on a deterministic per-range key checksum as the trustworthy at-rest check.
Proposal
On resume (and only on resume; fresh copies claim nothing), the sender checksums the key range the cursor claims was delivered: every prior table in copy order, plus the resume table through afterKey inclusive. Keys only, primary-key order, local-only records excluded, capped, wall-clock paced. It sends the per-table checksums together with the exact bounds and cap it used, right after COPY_START, as a new COPY_RANGE_CHECKSUM message. The follower computes the same checksums over the identical range (the payload's bounds, not a re-read of its own cursor, which may have advanced or been removed since the request) and compares.
Alert-only: a mismatch logs an error with exact per-table key counts and stashes the details on the connection. No forced re-copy (against receiver-side undecodable drops a re-copy loops forever) and no cursor surgery. Known benign-drift sources under live traffic (rows inserted or deleted in the range since the copy began, live deliveries from other sources, local writes) make small drift expected on busy tables; a poisoned cursor shows as a massive one-sided gap.
Old senders and receivers interoperate unchanged: the message id is new and the command switch ignores unknown ids.
Cross-references
- harper-pro#537 observed divergence; harper-pro#538 review thread (checksum agreed as the follow-up).
Lavinia, via Claude
Summary
A bulk-copy resume trusts the follower's persisted cursor: everything in copy order before
{currentTable, afterKey}is assumed delivered, and the sender skips straight past it. Nothing ever verifies that assumption against actual table content. When a cursor outruns delivery (the harper-pro#537 poisoned-cursor class, or any future variant), every resume re-seals the hole: the skipped range is never re-examined, the counts sit thousands of rows apart, and no signal fires anywhere.The reworked stream-count verification (harper-pro#538) covers in-flight loss within one copy session. It cannot see at-rest holes from earlier sessions, which is exactly where poisoned cursors live. In the #538 review we converged on a deterministic per-range key checksum as the trustworthy at-rest check.
Proposal
On resume (and only on resume; fresh copies claim nothing), the sender checksums the key range the cursor claims was delivered: every prior table in copy order, plus the resume table through
afterKeyinclusive. Keys only, primary-key order, local-only records excluded, capped, wall-clock paced. It sends the per-table checksums together with the exact bounds and cap it used, right after COPY_START, as a newCOPY_RANGE_CHECKSUMmessage. The follower computes the same checksums over the identical range (the payload's bounds, not a re-read of its own cursor, which may have advanced or been removed since the request) and compares.Alert-only: a mismatch logs an error with exact per-table key counts and stashes the details on the connection. No forced re-copy (against receiver-side undecodable drops a re-copy loops forever) and no cursor surgery. Known benign-drift sources under live traffic (rows inserted or deleted in the range since the copy began, live deliveries from other sources, local writes) make small drift expected on busy tables; a poisoned cursor shows as a massive one-sided gap.
Old senders and receivers interoperate unchanged: the message id is new and the command switch ignores unknown ids.
Cross-references
Lavinia, via Claude