Skip to content

fix(replication): close copy-cursor crash + recover wedged followers - #326

Merged
kriszyp merged 3 commits into
mainfrom
fix/321-tables-guard-and-cursor-cleanup
Jun 11, 2026
Merged

kriszyp merged 3 commits into
mainfrom
fix/321-tables-guard-and-cursor-cleanup

Conversation

@ldt1996

@ldt1996 ldt1996 commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Summary

Two surgical changes in replication/replicationConnection.ts that close the remaining two manifestations of harper-pro#321 not covered by #322's write-site guard.

#322 stops new malformed cursors from being persisted. This PR closes the silent crash variant and recovers clusters already wedged on a corrupt cursor.

What this fixes

(b) Silent channel death at L1712 — tables undefined throw

The resume-validity check at L1712 was tables[copyResume.currentTable]. On a freshly-joined peer, tables itself can be undefined, and tables[…] then throws TypeError: Cannot read properties of undefined (reading 'undefined') (char 55 of the original line). The throw is caught by the outer .catch at L1874-L1877 which calls close(1008, ...) — a silent subscription teardown: no warn log on the leader, no restart-loop visible, just a closed channel that takes the data direction with it. The colleague's bench captured exactly this stack signature.

Change: optional-chain tables to tables?.[copyResume.currentTable]. When tables is undefined the index returns undefined, tableToTableEntry(undefined) is falsy, and we fall into the existing warn-and-recopy branch (already correct: copyStartTime is preserved from copyResume?.copyStartTime at L1701, so the post-copy resume point doesn't lose its anchor). Same branch also picks up the malformed-currentTable case (a no-op cleanup behavior change since tables[undefined] was already falsy in that case).

(c) Recovery for already-wedged followers at L2420

A cluster that's already looping with a corrupt cursor on disk does not recover from #322's fix alone — that fix only prevents new poisoning. The persisted entry stays, gets sent verbatim in every subscription request, and the leader's warn-and-recopy branch runs every reconnect. In the production failure surface the colleague reported (spectrum-et, system DB 800M+, "67 MB SSTs landing each cycle but never finalizing") COPY_COMPLETE may never arrive while the loop is active, so the cursor never gets cleared.

Change: right after dbisDB.get([Symbol.for('copyCursor'), nodeId]), if copyCursor && !copyCursor.currentTable, warn, dbisDB.remove(...), and treat copyCursor = undefined for the rest of the flow. Next reconnect requests a clean full copy instead of forwarding the malformed cursor. One-shot recovery, no protocol bump.

What this does NOT touch

Verification

  • npm run build: dist emitted with both changes; no new TS errors. Pre-existing _socket/Numeric/code/isHandled errors unchanged.
  • npm run lint: pre-existing lint errors unchanged; only replicationConnection.ts mention is an unrelated subscriptionRequest never-assigned warning at L693.
  • Bench validation pending — would appreciate a run against this branch.

Test plan

  • Bench harness run against this branch (colleague's 500k records, CPU-constrained receiver, reciprocal subscription with empty side, single 8s cut at +6s)
  • Confirm previously wedged cluster recovers on next reconnect after this fix
  • Confirm Cannot read properties of undefined (reading 'undefined') no longer appears in error logs for peers with undefined tables

🤖 Generated with Claude Code

…(harper-pro#321)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@ldt1996
ldt1996 requested a review from a team as a code owner June 10, 2026 18:08
@ldt1996 ldt1996 changed the title fix(replication): close copy-cursor crash + recover wedged followers (harper-pro#321) fix(replication): close copy-cursor crash + recover wedged followers Jun 10, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request addresses replication issues related to malformed cursors and undefined tables. It introduces optional chaining when checking the copy-resume table to prevent runtime errors, and discards malformed copy-resume cursors on disk to trigger a clean full copy. The review feedback points out that while optional-chaining tables prevents a crash at line 1716, a similar TypeError could still occur later at line 1855 if tables is undefined, suggesting that this reference should also be guarded.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread replication/replicationConnection.ts
@claude

claude Bot commented Jun 10, 2026 •

Copy link
Copy Markdown
Contributor

Reviewed; no blockers found.

@heskew heskew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work running both manifestations down — I traced the logic and it holds up: tableToTableEntry(undefined) is falsy so the guard recopies instead of throwing, for…in over an undefined tables is a safe no-op, and copyStartTime is captured before copyResume is reset so the resume anchor survives. The disk-cursor cleanup matches the existing un-awaited remove, too.

Two small things, neither blocking:

  1. Include the repro test. You mention badCursorRepro.test.mjs already exists and exercises both branches deterministically (no dependence on the flaky natural #321 trigger). Since our standard is coverage for changed functions, that's exactly what belongs in this PR rather than left in the working tree.
  2. Trim a couple comments so they don't rot (suggestions inline) — a few pin moment-in-time details that'll drift.

Thanks for the detailed write-up; made this easy to follow.


🤖 Posted by Claude on Nathan's behalf

Comment thread replication/replicationConnection.ts Outdated
Comment thread replication/replicationConnection.ts Outdated
… trims, unit test)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@ldt1996

ldt1996 commented Jun 10, 2026

Copy link
Copy Markdown
Contributor Author

@heskew thanks for the careful trace — both items addressed in 1ab03b1:

Comments tightened with your exact suggested wording (no "silent" overclaim at L1712, no L1701 anchor in the comment after L1716).

Regression test added at unitTests/replication/discardMalformedCopyCursor.test.mjs. To make it cleanly unit-testable, I extracted the cleanup as discardMalformedCopyCursor near the top of replicationConnection.ts — matches the existing shouldTerminateIdlePing / createReceiveWatchdog export pattern and the selectSubscriptionNodes (#323) / isReusableConnection testing pattern. The call site at the read path is now a single helper invocation. 7 cases (well-formed passthrough, undefined currentTable, empty-string currentTable, nodeId undefined, dbisDB missing, absent warn callback) — all pass, and the full 52-test replication unit suite stays green.

Also picked up Gemini's L1855 finding in the same commit (tables.test → tables?.test) since it's the same defensive-optional-chain pattern in a debug log path that would otherwise still throw on a freshly-joined peer.

@kriszyp
kriszyp merged commit 0537ea5 into main Jun 11, 2026
27 checks passed
@kriszyp
kriszyp deleted the fix/321-tables-guard-and-cursor-cleanup branch June 11, 2026 03:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants