Count only a record's residents toward replicatedConfirmation, and refuse a count they cannot meet before writing

[kriszyp]

⊙ Problem

replicatedConfirmation: N resolves once N peers report an applied position for the database that is at or past the write. On a table with a residency list, that position is also passed by peers that do not hold the record: a non-resident peer receives only an invalidation stub, and a peer outside the list may receive nothing at all for that record. Either one still counts. A write can therefore report "confirmed by N" while no other node holds a complete copy (Count only peers that hold a complete copy toward replicatedConfirmation — harper-pro#1031). Core is the layer that knows each record's residency, but it never passed that to the replication layer.

❓ Your call: Is the requirement warranted? As specified, yes. The task owner ruled on 2026-10-11 that counting non-resident peers is a bug, not the contract. They also chose the per-record rule, fail fast when N exceeds the residents, and no protocol change. This PR implements those rulings without re-opening them.

💡 Solution

The commit now gives the confirmation callback a 4th argument, residentPeerLists: one list of other residents per distinct residency among the transaction's confirmable writes, with the writer and duplicate names removed. Harper-pro's waiter counts a peer only toward the lists it belongs to, and every list needs N confirmed members. A write with no residency adds no list, because N confirmations from any list already come from N distinct peers. When no write has a residency, the argument is undefined and the waiter counts as before.

• What it waits for. The confirmation position is now the committed transaction's reserved log key, which is what peers acknowledge. Before, it was the version of the last write's stored entry. A record version can sit below the log key (an explicit timestamp under the origin floor) or above it (a remote-origin version kept by a merge), and the stored entry can belong to an older write. Under setResidencyById, a write that excludes the writer stores no entry at all, so it was never confirmed.
• Which residents. The commit handler records the residencyId of the audit entry each write appended. A coordinated retry appends nothing and keeps it, so the residents counted are the ones the replicated entry names. Residents are resolved once per residency id. getResidencyId now never claims an id already in use, and it writes the id-to-list mapping before the set-to-id one, so an id that is discoverable always resolves to its one list.
• Nothing to confirm. A delete on a table with neither audit nor delete tracking appends nothing, so it is exempt (skipReplicationConfirmation). A commit made only of exempt writes neither waits nor runs the feasibility check. A commit whose writes all lost to newer entries in every round falls back to the stored-entry target.
• Refusals before anything lands. If N is larger than a record's other residents, the write's commit handler refuses it before staging anything (no record, no index entries, no blob reclamation), with a 400 such as Cannot confirm replication to 2 node(s) for record "x" in database "data": it is resident on 1 other node(s) (node-b). The refused write turns inert, so a caller that catches the error cannot commit it, even on a retry. Just before native submission, the commit rechecks the lists (confirmation may have been requested after a write was saved). It then runs a feasibility check the replication layer can now register with replicationConfirmation(callback, feasibility). Harper-pro uses it for its recipient bound, which used to answer 400 for a write that had already landed. An error after the native commit becomes a rejected completion, so the commit's bookkeeping (blob cleanup, lock and context release) still runs.
• Count-based placement on update. replicateTo: N built an update's placement as [self, ...previousResidency], and the stored list already starts with the writer. Each update therefore repeated the writer and shed a replica: [S,A,B] became [S,S,A], then [S,S,S]. This was a pre-existing data-placement bug, which the new refusal surfaced as a 400. The carried-over list now leaves this node out.

⚖️ Alternatives

• Framing rechecks: better-alternative-exists five times; each was adopted, and none cleared. The review CLI required a recheck after round 5. Each verdict named a smaller or safer variant inside the same design, and I adopted every one: target the committed log key, not record versions; claim residency ids atomically (the resolved lists are cached); refuse through a pre-submission feasibility hook on both submission paths; exempt entry-less deletes and marker-only commits; key the target off the sticky appendedAuditEntry; refuse and observe an asynchronous feasibility check; reuse the checked lists. The last verdict said its points "do not require replacing the design", so I adopted them without a sixth round. The original plan review cleared with Framing-Verdict: chosen-approach-sound.
• Receiver-side, per-record acknowledgements. Rejected: this is a protocol change. Receivers on older versions would never send the acknowledgement, so confirmed writes would time out during a rolling upgrade. The writer already knows each record's residency.
• Harper-pro re-reads residency from the committed entry. Rejected: that read happens after the native commit. A concurrent overwrite by another worker can replace the entry's residencyId by then. The entry also does not exist when the writer stored no local record.
• Use the native transaction's logEntries. Rejected: RocksTransactionLogStore fills that array only when an aftercommit listener exists. It also skips appends on a retry, so it can be absent or describe a different attempt.

❓ Your call: No benchmarks. The new costs on writes that don't request confirmation are recording a residencyId on each placed, audited write and a synchronous atomic id claim when a new residency list is first minted. A confirmed commit builds its resident lists once and looks residents up through a cache that is cleared past 10,000 ids. The review flags these as unmeasured. I judged them negligible next to the encode, native put and audit append each write already does. A placement-churn workload that keeps crossing the cache bound would be the first thing to measure.

❓ Your call: A refused write is refused per write, and the rest of the transaction can still commit if the caller catches the error. The alternative is to poison the whole transaction on refusal, which is simpler and fails closed. The per-write choice is why the refused write has to stay inert on retry rounds. Switching later changes what callers see.

❓ Your call: A table source's before hook (source.put/patch) runs before the commit handler, so a write-through source can receive a write that Harper then refuses. Record-based placement of a patch needs the merged record, which only the commit handler has. A source hook also already precedes every other commit-handler failure (conflict abort, record-lock refusal), so the write-through contract is unchanged. Documented in resources/DESIGN.md.

❓ Your call: LMDB is untouched. harper-pro never registers LMDBTransaction's separate confirmation callback, so replicatedConfirmation on LMDB has never waited (fullyConnectedReplication.test.mjs says as much). LMDB parity belongs with fixing that registration, in a separate change.

🔧 Changes

• resources/DatabaseTransaction.ts
  • confirmationTarget picks the committed log key, else the stored-entry fallback, deciding by whether a confirmable write appended to the committed handle (appendedTo). ⚠️ Look hardest here: this is the position peers must acknowledge.
  • residentPeerLists and confirmingResidents build the per-residency lists. Writer and duplicates are removed, and the lists come from a bounded per-database cache of frozen lists.
  • refuseUnconfirmable runs before both native submissions: the iterator-replay path and the ordinary path. It runs the feasibility check registered with replicationConfirmation(callback, feasibility), and aborts on refusal.
  • A post-commit error becomes a handled rejected completion, so blob cleanup and lock/context release still run.
• resources/Table.ts
  • The _writeUpdate commit handler refuses an unmeetable count before staging and makes the refused write inert. It records the audit entry's residencyId, kept across coordinated retries.
  • Count-based placement keeps the writer out of the carried-over residency. This fixes the pre-existing replica loss on update.
  • getResidencyId claims a new id and both mappings in one transaction.
  • A losing delete is marked skipped, and an entry-less delete is exempt from confirmation.
• resources/DESIGN.md gets the invariant note, indexed from the root DESIGN.md.

✅ Verification

• unitTests/resources/residencyConfirmation.test.js (19 tests, RocksDB) covers:

  • lists for record-based residency, replicateTo, and mixed/duplicate residencies;
  • pre-staging refusal with no record and no index entry;
  • caught refusal, including an existing key;
  • late activation refused before submission;
  • the committed-log-key target below the origin floor in both write orders, plus deletes;
  • the losing-delete and nothing-to-confirm fallbacks;
  • count-based placement across three updates;
  • the feasibility hook (called pre-submit; a refusal writes nothing; an async check is refused with no unhandled rejection);
  • BigInt key naming, omitted-writer writes, and late-requested omitted-writer updates.

  The new tests discriminate: they fail on the base sources (7 of 8 originally) and on each superseded intermediate target.

• npm run test:unit:resources: 4151 passing. npm run test:unit:main (isolated HOME): 6877 passing. Under HARPER_STORAGE_ENGINE=lmdb, writerResidency and reloadMarker pass, covering the atomic id claim on LMDB.

• End to end: harper-pro's new three-node integrationTests/cluster/residencyConfirmation.test.mjs against this core passes 3/3, and fails on base. The existing confirmation suites fullyConnectedReplication, replicationTopology, nonReplicatedDatabase, selectiveTableSubscription, removeNodeBlastRadius and replicationWorkerPool pass.

• After the PR opened, two more fixes landed. The pre-submission check now runs only before a handle's first submission: a retry of that same handle (isRetry) is not re-checked, and a fresh handle committing while the outer commit is in flight is. Resident lists count only writes that appended to the submitted handle. The suites below were re-run on the final head.

• A pre-existing flake, unitTests/security/userRecordLookups.test.js ("onUserChange re-subscribes…"), fails about 1 in 3–6 runs on base too.

Refs HarperFast/harper-pro#1031. Companion: harper-pro PR (bumps core to this branch). After merge, documentation#723 needs to describe this contract.

_{🤖 Generated by Claude (Anthropic, Opus 5.5); posted via @kriszyp.}

Related PRs: #2918 overlaps (DatabaseTransaction save() arguments and the pre-submission abort block), #3092 overlaps (Table source-hook ordering before the commit handler), 3 others independent
Complexity: complicated

_{Review-Coverage: authored=claude; ran=cursor-composer,gemini,codex,cursor-muse; adjudicated=domain; declined=cursor-grok,cursor-kimi; rounds=11; full=5 @ a8c51af}

_{Review-Attention: deep ~40m (critical: DatabaseTransaction.ts, Table.ts; decisions: do-less-alternative, caught-refusal-semantics, refuse-before-landing, sync-only-feasibility, lmdb-unchanged) @ a8c51af}

[gemini-code-assist]

Code Review

This pull request implements changes to ensure that replicatedConfirmation counts only the actual residents of each written record. It introduces caching of residents by residency ID, adds a feasibility check before native transaction submission to refuse unconfirmable writes early, and updates the transaction commit process to track the exact log keys and resident peer lists. A high-severity issue was identified in DatabaseTransaction.ts where accessing this.writes[0] directly could throw a TypeError if the first write was detached; it is recommended to safely find the first non-null write instead.