Count only resident peers toward replicatedConfirmation

[kriszyp]

⊙ Problem

replicatedConfirmation: N counts any peer whose applied position for the database has reached the write. On a table with a residency list, that includes peers that do not hold the record. A non-resident peer is sent only an invalidation stub, and a peer excluded by replicateTo (implemented as residency) may be sent nothing for that record. Both still pass the database position. A write could resolve "confirmed by N" while no other node held a complete copy. Refs #1031.

❓ Your call: Is the requirement warranted? As specified, yes. The task owner ruled on 2026-10-11: count only peers that hold a complete copy; a record needs N confirmations from its own residents; fail fast when N exceeds them; no protocol change. documentation#723 documents the old any-N behavior and is on hold until this lands. It needs to be updated to this contract after merge.

💡 Solution

Core (Count only a record's residents toward replicatedConfirmation — harper#3148) now passes the confirmation callback one list of other resident peers for each distinct residency among the written records, and targets the committed transaction's log key. createConfirmationWaiter resolves only when every list has N confirmed members. A peer still counts once, by identity (harper-pro#633), and the alreadyConfirmedPeers seed goes through the same check. The writer decides who is eligible and reuses the existing COMMITTED_UPDATE acknowledgement, so receivers of any version need nothing new. When there are no lists (a table without residency), the waiter keeps the database-wide count and the getReplicationRecipients bound as before. The recipient guard moves into core's new pre-submission feasibility hook (assertConfirmationFeasible), so its 400 arrives before the write lands rather than after it. It also checks each list against this node's recipients (assertResidentsReachable): a record homed on a node that is no longer a cluster member, or on a peer this node does not send the database to, is refused at once instead of waiting out the 900 s timeout.

Before → after, with three nodes W, B, C, a record homed on [W, B], B down, and X-Replicate-To: *;confirm=1:

	before	after
C applies the invalidation stub	write confirmed	still waiting for B
B comes back and applies the record	n/a	write confirmed
confirm=2 on that record	waits up to 900 s, then 504	immediate 400 (resident on 1 other node(s) (B)), nothing written
record homed on [W, B, retired-node], confirm=2	waits up to 900 s, then 504	immediate 400, nothing written

⚖️ Alternatives

• Planning review: Framing-Verdict: chosen-approach-sound. The core framing recheck (adopted there) moved this PR's recipient guard into core's pre-submission feasibility hook.

Weighed in the core PR: per-record acknowledgements from receivers would need a protocol change and break rolling upgrades. Having pro re-read residency from the committed entry would race concurrent overwrites. Both were rejected.

❓ Your call: Each confirmed transaction with resident lists runs assertResidentsReachable, an allocation-free scan of residents × this node's recipients. Its size is bounded by the cluster, and no measurement was taken. The pre-push review flags it as an unmeasured hot path; I judged a benchmark unnecessary at cluster-sized inputs.

❓ Your call: For unrestricted tables the waiter now settles at confirmed.size >= N rather than === N. For integer N nothing changes. A fractional count such as confirm=1.5, which REST accepts, used to wait out the timeout and now settles at 2 confirmations.

🔧 Changes

• replication/knownNodes.ts
  • createConfirmationWaiter takes residentPeerLists. Its predicate resolves once every list has N confirmed members, without allocation (countMembers). The alreadyConfirmedPeers seed goes through the same predicate. ⚠️ Look hardest here: this is where a non-resident's acknowledgement stops counting.
  • assertConfirmationFeasible is the recipient bound plus assertResidentsReachable. It is registered as core's pre-submission feasibility check, so its 400 arrives before the write lands.
• replication/DESIGN.md: item 29 and a test-table row.
• core: bumped to the harper#3148 branch head.

✅ Verification

• integrationTests/cluster/residencyConfirmation.test.mjs with the fixture (config.yaml, resources.js, schema.graphql): three nodes, writes through REST. The operations API runs on the main thread, which loads no application setResidency.

  1. confirm=2 on a record homed on [W, B], and on [W, B, retired-node]: immediate 400, and nothing is written.
  2. With B stopped, W writes three things, each with confirm=1: a record homed on [W, B], an Open record with X-Replicate-To: B, and a two-record PUT ([W,B] + [W,C]). The test waits until all are committed on W. A later probe homed on [W, C] confirms, which proves C applied them. All three stay unconfirmed for 3 s. After B restarts and rejoins the mesh, all three confirm, and B holds the full records.

  Passes 3/3 in repeated runs. On base it fails both tests: all three writes were confirmed with B down, and confirm=2 was accepted with 204.

• unitTests/replication/confirmationWaiter.test.mjs: 33 passing. Covers non-resident ignored, seed filtered, per-record lists, one peer counted toward several lists, shortfall message, [] treated as database-wide, and assertResidentsReachable.

• npm run test:unit: passing. Existing confirmation cluster suites unchanged and passing: fullyConnectedReplication, replicationTopology, nonReplicatedDatabase, selectiveTableSubscription, removeNodeBlastRadius, replicationWorkerPool.

_{🤖 Generated by Claude (Anthropic, Opus 5.5); posted via @kriszyp.}
Depends-on: HarperFast/harper#3148

Related PRs: #1030 independent (set the base core pin this bumps from)
Complexity: complicated

Origin — the dispatch brief this PR was written from

replicatedConfirmation counts only peers that hold a copy of the record

Fix harper-pro#1031: replicatedConfirmation: N must count only peers that hold a complete copy of the written record(s). Today createConfirmationWaiter in replication/knownNodes.ts counts any peer whose per-database applied watermark (CONFIRMATION_STATUS_POSITION) reaches the write's txnTime, so a non-resident peer (received only an INVALIDATED stub) or a replicateTo-excluded peer (received nothing) is counted. This is a coordinated core + pro change: core supplies the per-write resident peer set to the replicationConfirmation(...) callback; pro's waiter ignores peers outside it. Use the coordinated-PR flow (harper-pro worktree with its own core submodule checkout; harper PR + harper-pro PR with the bumped core pointer). Open both as drafts.

Acceptance

• Test (fails on current main): record-based residency list where the only non-writer resident is down; a write with replicatedConfirmation: 1 is not confirmed by a non-resident or replicateTo-excluded peer that has advanced its watermark.
• Test: N exceeding other residents fails fast with a clear error.
• Test: multi-record transaction with differing residency applies the per-record rule.
• Existing confirmation unit/integration tests for unsharded tables pass unchanged.
• Both PRs draft; harper-pro PR bumps core to the harper PR's commit; PR bodies reference harper-pro#1031 and note documentation#723 needs updating after merge.

_{Dispatch: task hp-1031-residency-aware-confirmation · queued by kris-session · ran by claude/opus/high · worker kzyp-xps-1}

_{Review-Coverage: authored=claude; ran=cursor-composer,gemini,codex; adjudicated=domain; declined=cursor-grok,cursor-kimi,cursor-muse; rounds=11; full=5 @ d1cf078}

_{Review-Attention: study ~20m (critical: knownNodes.ts; decisions: per-residency-each-meets-count, pre-submission-feasibility-hook, direct-recipients-only, writer-side-eligibility, do-less-alternative) @ d1cf078}

[gemini-code-assist]

Code Review

This pull request updates the replication confirmation logic to ensure that replicatedConfirmation only counts peers that are resident for the written records. It introduces resident peer lists tracking in createConfirmationWaiter and adds pre-submission feasibility checks to refuse impossible confirmation counts early. The changes are accompanied by comprehensive unit and integration tests, as well as updated design documentation. Feedback on the changes highlights a potential TypeScript compilation error under strictNullChecks in replication/knownNodes.ts where residentPeerLists is mapped directly without being narrowed; using the already-narrowed residentGroups variable is suggested to resolve this.