Cluster record locks: operator-agreed home map transport and successor-freshness barriers (harper-pro#825, harper#2542 inside #822)

[kriszyp]

Cluster record locks over the operator-agreed home map (harper-pro#825 inside #822), now with successor freshness: a handoff carries exclusion and proof that the successor has applied its predecessors' writes. Core's half is harper#2613 (lineage on the release entry), harper#2625 / PR harper#2627 (the lockBarrier fence, writeLockBarrier, deadlineMs) and harper#2628 / PR harper#2630 (the awaited apply-failure listener) — all merged; this PR's core submodule is pinned at harper main 1c312feb4. Default off, unchanged.

Why the scope changed again

harper#2613 made ClusterLockTransport.establishLockFreshness() required, so this branch stopped compiling against core. The transport half of §7 (docs/record-lock-ownership.md) is what this round adds. Six planning rounds (replication/RECORD_LOCK_FRESHNESS_DESIGN.md, revision history) each found a real hole in a numeric-watermark design; the mechanism that survived is the reviewer's own do-less from round 5: prove every unsatisfied cross-origin dependency with an exact, same-table, nonce-bearing lockBarrier entry and nothing else.

What this round adds: successor freshness

• replication/recordLockFreshness.ts — the barrier. For each inherited (origin, position): refuse outright a non-member, a peer at another capability level, an unreplicated table, a poisoned (origin, table), an invalid position, or this node's own lineage after a reclone; otherwise register a nonce, ask the origin for a barrier, and resolve only when the exact (origin, position, nonce) entry has committed here. Recovery (null) does the same for every home-map member and returns the pairs. Every wait is bounded by min(deadlineMs, MAX_LOCK_LEASE_MS), capped per database, settled exactly once, and closed with the transport.
• replication/recordLockRpc.ts — record_lock_barrier: node principal → current member at the exact level → table replicates → nonce → per-caller token bucket → one writeLockBarrier per request, never merged across callers.
• replication/replicationConnection.ts — a lockBarrier record is captured at decode and reported from its frame's onCommit (relayed to the coordinating thread when applied elsewhere); every record this node drops poisons (origin, table) durably before the drop completes, and a poison write that fails holds the frame; a clone attempt writes the ever-recloned flag before its first row; the peer's exact level is recorded at handshake.
• replication/recordLockPoison.ts — poison and the reclone flag in the database's dbis store; poison is permanent (a base copy is a put snapshot and cannot re-deliver a dropped delete).
• replication/recordLockTransport.ts — both transports implement the method; slot 31 carries the exact level; cluster_status.recordLocks reports barrier counters and poisoned pairs; core's apply-failure listener (harper#2628) is registered per database and unregistered on release.
• Capability level 3 → 4; replication.recordLocks resolves to false on LMDB with one error line.

The home map (harper-pro#825): why that scope changed

harper#2498 merged with ClusterLockTransport.epoch() replaced by homeMap(database) returning an operator-agreed LockHomeMap { generation, homes[], homeIncarnation } — core's own design doc for it says the map is "supplied by harper-pro; core never computes it and never advances it." There is no epoch() shape left to build the old static-epoch scaffolding against, and the static epoch was itself the design's open blocker (two nodes independently deriving different rings for the same key — two arbiters). Rather than improvise an interim shape, this branch went needs-input; the task owner's answer was to build harper-pro#825's real mechanism now, inside this PR, tracking core at current origin/main.

Because homes are now operator-agreed (never derived from hdb_nodes membership or liveness), the earlier "two rings" hazard is structurally gone: every node reads the same durably-staged/activated generation instead of computing its own view of the ring.

What the home-map transport is

Core (merged, origin/main) owns the coordinator — the home ring, the delegation table, recall-and-drain, fencing tokens, and the one control entry (lockRelease) still on the transaction log. This PR supplies everything core cannot know: the durable operator-agreed generation state, the wire, and the transport.

• hdb_record_lock_homes (replication/recordLockHomes.ts) — a dedicated LOCAL_ONLY system table (getRecordLockHomesTable, never replicated or LWW-merged), one row per database holding {active, staged, highestActedOn, fenced[]}. Three super_user-gated operations transition it: record_lock_stage_generation atomically retracts active in the same durable write that stages g+1 (so a successful response IS quiescence evidence, not a race against a later refresh), record_lock_fence_external records an operator's external attestation that an unreachable node was stopped, and record_lock_activate_generation promotes staged → active after the operator has externally collected a stage response from every affected node and waited DELEGATION_LEASE_MS + LOCK_LEASE_SKEW_MS — no node ever measures that elapsed time itself. digestOf is a length-prefixed SHA-256 over (generation, homes[]), not delimiter-joined (so ['A','B'] and ['A\0B'] can't collide) and not a truncated generation >>> 0 (so generation 1 and 2**32+1 can't collide). Same-database stage/fence/activate calls serialize behind an in-process queue (withRow) so a read this call's decision is based on can't go stale before the write.
• homeMap(database) (replication/recordLockTransport.ts) — a frozen, per-thread cache of this node's active generation, refreshed only on change, returning undefined until this node has an active generation AND every named peer's advertised digest agrees (a mismatch fails the whole map closed, never shrinks the ring). Three independent triggers rebuild the transport object so core gets a fresh lazily-built coordinator at the instant each becomes true: the first-ever known home incarnation, ownership newly conferred, and an active generation first appearing or changing — each is a distinct restart-quarantine waiver (grantableAfterMono) or fencing-seed gap in core's coordinator that only closes if the coordinator is built after the fact, not before.
• The digest wire (replication/replicationConnection.ts, replication/protocolCapabilities.ts) — recordLocks capability bumped to level 3 (the old static-epoch level 2 never shipped enabled); a new RECORD_LOCK_HOMES_DIGEST frame and a second shared-status-buffer slot (RECORD_LOCK_HOMES_AGREEMENT_POSITION) carry per-peer digest agreement, sent only after capability discovery so a pre-upgrade peer never receives a frame it hasn't advertised support for. Agreement is reconciled centrally by database (lastPeerDigest), independent of which of the mesh's separate per-direction connection objects last touched it.
• A fail-closed, cross-thread change-acknowledgement protocol (replication/recordLockTransport.ts) — a stage/activate response is real quiescence evidence only if every HTTP-worker thread (not just the durable write) has applied the change; a missing or negative acknowledgement now fails the operation (503) rather than being logged and treated as success, and the origin-to-main relay carries a longer budget than the leaf fan-out it waits on. An idempotent retry (the noop decision branch) still re-notifies, so a retry after a failed relay actually reconciles the gap it exists to close.
• requestDelegation / recallDelegation / owner relay (replication/recordLockRpc.ts) — unchanged in shape from the prior tranche: the requester's identity is the authenticated node principal, never a payload field; a relay through main that times out answers not-home, never a grant.
• ownsCoordination(), per-database owner-worker assignment, cluster_status.recordLocks — unchanged in role; recordLockOwnerFor's async-handoff/PENDING_BUMP behavior and setRecordLockOwnership's transport-recreate-on-first-ownership are both from the prior tranche.
• A disabled transport now answers 503, not a generic 500 (replication/recordLockTransport.ts) — createDisabledRecordLockTransport throws a ClientError(..., 503), matching the status code the rest of the transport uses for "not available yet."
• A startup warning (replication/recordLockTransport.ts) fires when replication.recordLocks is enabled, naming harper#2542 and the measured 0.05–0.13% silent-lost-update rate — operator-visible at the point the switch takes effect, not only in this PR's own docs.
• replication/RECORD_LOCK_HOMES_DESIGN.md (new) — the full design note, including the revision history of two rejected planning-review rounds (see below) and the "For the human reviewer" items carried into this PR body.
• The §10 cost measurement (replication/RECORD_LOCK_COST_DELEGATIONS.md, replication/RECORD_LOCK_COST_BASELINE.md, raw runs under README, run 1, run 2, run 3, run 4 (starvation)) landed on this branch from a prior measurement PR and predates the home-map redesign; kept as-is rather than re-run (it measures against a core pin that has since moved, so its numbers are stale relative to this PR's current pin — a re-measurement is future work, not something this PR redoes).

The home-map design path (two rejected planning-review rounds)

The §4.3 stage→quiesce→drain→activate transition went through mandatory planning review three times before landing:

• Round 1 (better-alternative-exists): rejected a "trusted orchestrator + canonical artifact" design — 4 blockers.
• Round 2 (better-alternative-exists): rejected the local-timer reduction that followed — a purely node-timed mechanism is unsound (4 concrete counterexamples); round 2's own stated recommendation — stage retracts immediately, activate is a separate operator-timed call — is what shipped, verbatim, not a third new mechanism, so no third planning round applied.

The chosen design deliberately does not build a machine-verified barrier for the drain wait: every node checks consistency (does this activation match what I staged), never elapsed time, because a self-measured wait is unsound across a restart or clock correction. The operator's own external wait (DELEGATION_LEASE_MS + LOCK_LEASE_SKEW_MS) is what makes activation safe — nothing in the code enforces it. Full history in RECORD_LOCK_HOMES_DESIGN.md.

For the human reviewer

• Decided, not open: a home-map transition is authorized by node identity alone, and that is accepted for this release. record_lock_apply_homes is requiresSuperUser; the per-node hop it fans out on is not — executeTransition registers without it and admits any principal principalNodeName resolves. The replication socket compounds it: replicationConnection.ts dispatches as server.operation(data, {user}, !isAuthorizedNode), so runWithOperationAuthorizationBypass is on and the nominally requiresSuperUser operations are reachable that way too. One compromised node — or one super_user on any single node, through the node-identity gap listed below — can stage and then activate a map of its choosing on a peer, and the digest check narrows that rather than closing it (homeMap() iterates its own active.homes, so a node rewritten to a singleton never consults a peer, and a peer fails closed only once the changed digest reaches it). The task owner ruled to accept it here, on three facts now written into RECORD_LOCK_HOMES_DESIGN.md → "Auth for the hop": a node principal is already trusted to write replicated data on every peer; this relay does not widen the authority, because the per-node operations were already reachable that way before harper-pro#862 and the relay only adds local re-validation; and nothing shipped is exposed, since the feature is off by default and inert until a generation is activated. What would close it — an operator-delegated proof carried on the transition — is harper-pro#869, a prerequisite for recommending this feature in production and for Record locks: decide what it would take to enable replication.recordLocks by default #853's default-on question. The dispatcher's node-wide verifyPerms bypass is wider than record locks and wants its own assessment; Record locks: a home-map transition is authorized by node identity alone, with no proof an operator asked for it #869 says so rather than folding it in. An operator who sets replication.recordLocks: true is now told the caveat at startup, next to the existing enablement warning, rather than only in the design note.

• Planning-gate recheck, round 32: Framing-Verdict: better-alternative-exists, carried rather than adopted. The pre-push CLI required a --mode plan recheck; the artifact is RECORD_LOCK_HOMES_DESIGN.md. It affirms the operator-stated map over failure-driven consensus and affirms keeping topology in harper-pro rather than core, but argues the option set missed a better implementation layer: a main-thread control-plane API with explicit topology and owner-lifecycle state, rather than HTTP workers reconstructing either. That is the same root cause as two of the carried majors (recordLockOwnerFor collapsing main/pending/none to undefined, and the proposal reader scanning raw hdb_nodes off the main thread). It is adoptable and would remove that class of finding rather than patch it — but it is multi-day work across subscriptionManager, recordLockTransport and the proposal path, so it is recorded here rather than started under a review round.

• Round 6 of the planning gate was resolved by overrule, not cleared. Its blocker claimed replay reorders a log by key so a post-rollback barrier (key 101) precedes an inherited write (key 200). A probe on the pinned rocksdb-js 2.9.0 shows per-log range reads are append-ordered (100, 200, 101, 300 appended → start>50 yields exactly that). The framing's incarnation-qualified append sequence (a wire/storage migration) is therefore not required. What the probe did surface is filed as harper#2629: on a resume, entries below the subscriber's cursor are never delivered — a pre-existing replication hazard on clock rollback; for locks it fails closed.

• Every hole class the design names is now recorded: harper#2628 merged (PR harper#2630) and listenForApplyFailures registers core's awaited listener per database, so a transaction core skips after a terminal apply failure poisons (origin, table) before the next event is consumed. The feature stays default-off pending the carried items below.

• Rollback: disable replication.recordLocks everywhere and restart before downgrading; retained lockBarrier entries replay into a level-3 sink as "malformed control entry" warnings.

• Coalescing is client-side and same-turn only, by design: a caller must match its own nonce, and a request sent before a caller's dependency was formed cannot prove it.

• Pre-push review, this round (codex + gemini + cursor-composer + domain, full, then delta): five majors in the freshness delta were found and fixed before push — a per-frame closure allocation on ordinary replication, a failed poison write cached as recorded, per-worker poison caches invisible to the coordinating thread, no poison recheck when a barrier settled, and the old barrier outliving a transport replacement. Majors that predate this round and are carried, not fixed: a transient storage error in the one startup refreshCache (or a rejected bumpHomeIncarnation) latches homeMap() undefined until an operator re-stages; homeMap() is on the per-acquisition path and walks the home set with a getDatabases() lookup (the design's "zero-allocation hit" is the cached delegation, not the map read); a lockRelease met before NODE_NAME on a fresh socket is skipped and its cursor advanced.

• Carried from before: RPC node identity is "authenticated user named like a node in server.nodes" (see Known gaps). Two items that were carried here are now closed rather than carried — withRow takes a node-scoped hold lock on the hdb_record_lock_homes row itself, so the per-isolate queue is no longer what excludes concurrent transitions, and harper-pro#865 / harper#2667 relay an off-owner lock() to the coordinating worker, so threads.count > 1 is served rather than 503'd.

Carried from the home-map rounds:

Decisions that are yours to make, not mine to have picked silently (full reasoning in the design note's decision ledger):

• Ship before harper#2542 (successor freshness) lands, opt-in and explicitly warned, vs. hold enablement. Measured silent lost updates (0.05–0.13% of sections at 3 contenders) remain either way; shipping now is reversible only while no operator has built on the primitive.
• The transition's safety rests on an external human wall-clock wait that no code measures, vs. a machine-verified barrier. A mistimed activation is a two-holder with no error anywhere. Reversing this means rebuilding the consensus mechanism the two rejected planning rounds above deliberately did away with.
• An unacknowledged worker now 503s the whole stage/activate call, even though the durable write (which already retracted active) has landed. The operator gets an error on an operation that half-succeeded and must retry to converge. Correctness is right; whether that's the operability you want is open — making the write itself conditional on the fan-out is a protocol change, not a constant.
• Staging still quiesces the whole database's cluster locks, not just the keys whose home is moving. The duration is no longer a fixed 365 s — stage drains and reports, so an operator with a proven-clean drain everywhere activates immediately (below) — but the window is still database-wide rather than scoped to the relocating keys. Narrowing that is a protocol change, not a tuning knob; it is the remaining lever on harper-pro#856.
• Multi-worker support is now claimed, not warned around — both gaps that made it a decision are closed, and what is left is how much you want to trust untested breadth. withRow takes a node-scoped hold lock on the row and writes back through that same locked handle, so a stage and a fence_external on two worker isolates serialize at the storage layer rather than at a per-isolate map (unitTests/replication/recordLockHomes.test.mjs, "a transition waits for a node-scoped hold on its row taken outside withRow"); and harper-pro#865 with core harper#2667 relays an off-owner lock() acquire/release to the coordinating worker, so a non-owner worker serves rather than 503s. Every cluster fixture still runs threads.count: 1 except the harper-pro#852 suite, so the multi-worker path has unit and targeted-integration coverage, not the full matrix.
• A departing node may or may not be activated, and the two operator paths disagree. A node in quiesce but not homes(g+1) is leaving the ring. record_lock_activate_generation accepts an activate on it — nothing in planActivate requires self ∈ homes, and neither does homeMap() — which leaves the leaver routing its own lock() to the new ring instead of answering 503. record_lock_apply_homes never sends it one (role: 'departing', activate: 'skipped'), so under that path the leaver cannot lock at all. Both are safe (a node outside homes is home to no key either way), so this is a usability contract, and RECORD_LOCK_HOMES_DESIGN.md §2 now states both rather than implying one. Picking one and making the other match is yours.
• homeMap() re-derives peer capability and digest agreement on every call rather than reading an invalidated pointer updated on change — O(homes) shared-buffer reads plus one allocation per lock acquisition, against a design that otherwise advertises the delegation hit as zero-message and zero-allocation. Buys freshness; reversible, but touches the core-facing interface shape.

Known gaps, not fixed in this PR, code-traced not demonstrated:

• The drain's quiescence proof has a mid-sweep hole, fix in flight in core and not yet pinned here. Core's quiesceDelegations swept one up-front snapshot of its live coordinators. The sweep awaits network recalls, so a transport replacement lands inside it: the successor adopts the predecessor's grants via handOffTo, the predecessor then closes as handed-off (which deliberately parks nothing in retiredCoordinators, because the grants are the successor's now), and the sweep sees an emptied predecessor and never the successor — { complete: true, outstanding: [] } on a database still admitting. That is exactly the result provesQuiescence() treats as authority to activate immediately, so it would become a double grant across a membership change. Found by this PR's final-artifact review. The pinned core still has it — quiesceDelegations (core/resources/recordLockCoordinator.ts:2789) still filters one up-front snapshot of liveCoordinators, and close() on a handed-off coordinator returns before parking anything in retiredCoordinators, so a successor created mid-sweep is in neither structure. No core change for it is open; it needs one (a fixpoint over the live registry). Until then, use the timed drain interval, not the drain result — provesQuiescence() is the only thing that licenses skipping the interval, and this is the way an empty sweep can still lie.
• replication/replicationConnection.ts:5436-5441 — a lockRelease skipped because peerCapabilitiesLearned is still false on a fresh socket takes skipAuditRecord(), advancing the peer's cursor past it, so it's never redelivered — the home can't re-grant until DELEGATION_LEASE_MS elapses, worse than the neighboring comment's stated "costs a requester a 423."
• replication/subscriptionManager.ts:771-777 (via recordLockOwnerFor) — an in-flight ownership handoff and "main owns" both surface as undefined from recordLockOwnerFor, so a subscription in flight during a handoff briefly places itself on the main thread and applies off-owner until reconcileWorkers corrects it.
• replication/recordLockRpc.ts:142-150 — node identity for the delegation RPC is "the authenticated username appears in server.nodes," so a super_user who creates a standard user named after a peer can mint or drain that peer's delegations through the operations API.
• replication/clusterStatus.ts:40-50 — the worker-side cluster_status wait has no deadline; a throw inside the now-async main-side status resolver leaves it pending forever and leaks the resolver entry.
• integrationTests/cluster/recordLockCluster.test.mjs — the "§4.3 stage/activate transition" suite's inline cleanup after a 503 assertion isn't in a finally, so one real assertion failure can mask itself as two; and startNode/several fetch helpers are duplicated near-identically against recordLockCost.bench.mjs (clusterShared.mjs is where the suite already keeps shared helpers).
• Test/integration coverage claimed in the design note is narrower than what shipped — corrected in this PR (see Changes below): digest mismatch, record_lock_fence_external, and restart/incarnation-ordering are unit-level only, not exercised against real IPC or a real cluster yet.

Changes

• The barrier — replication/recordLockFreshness.ts (new): refusals, one nonce-registered request per dependency, exact (origin, position, nonce) matching in both arrival orders, poison rechecked at settle, recovery across every member, deadline/cap/close lifecycle; unitTests/replication/recordLockFreshness.test.mjs (new) covers each.
• Durable poison and the reclone flag — replication/recordLockPoison.ts (new): store-backed, fail-closed reads on the cold path; a failed write is retried, never cached as recorded.
• The barrier RPC — replication/recordLockRpc.ts: record_lock_barrier, checks before state, one writeLockBarrier per request, per-caller token bucket.
• Transport — replication/recordLockTransport.ts: both transports implement establishLockFreshness; slot 31 exact level; applied-barrier relay to the coordinating thread; core's apply-failure listener consumed when present; status counters and poisoned pairs; barrier closed on recreate/release.
• Receive path — replication/replicationConnection.ts (recordReplicationHole, one per connection), barrier capture at decode, report from onCommit, ever-recloned at COPY_START, exact level recorded at handshake.
• Capability and gate — replication/protocolCapabilities.ts (level 4), replication/recordLockConfig.ts (LMDB refusal).
• Core pin — the core submodule: harper main 1c312feb4 (harper#2627 lockBarrier/writeLockBarrier/deadlineMs; harper#2630 registerReplicatedApplyFailureListener).
• Docs — replication/RECORD_LOCK_FRESHNESS_DESIGN.md (new: the design, six planning rounds, the probe, the overrule), replication/DESIGN.md, replication/RECORD_LOCK_HOMES_DESIGN.md, replication/RECORD_LOCK_COST_DELEGATIONS.md (convergence and recovery rows).
• Tests — integrationTests/cluster/recordLockCluster.test.mjs (exact 1..N, barrier-applied and no-poison checks), unitTests/replication/recordLockTransport.test.mjs (slot 31, disabled transport, barrier wiring).

From the home-map rounds:

• Generation state and its operations — replication/recordLockHomes.ts (new): the hdb_record_lock_homes table, canonicalization/digest, planStage/planActivate pure decisions, withRow serialization, and the three registered operations.
• Transport and cross-thread protocol — replication/recordLockTransport.ts: homeMap(), the three transport-recreate triggers, the fail-closed ack protocol, digest reconciliation, and the 503-not-500 disabled-transport fix.
• Wire — replication/replicationConnection.ts (the RECORD_LOCK_HOMES_DIGEST frame, sent only post-capability-discovery), replication/protocolCapabilities.ts (level bump to 3), replication/knownNodes.ts (the second shared-status-buffer slot).
• RPC, config, status, shared threading — replication/recordLockRpc.ts (a generation rename from the old epoch field, a guarded worker-side postMessage), replication/recordLockConfig.ts, replication/clusterStatus.ts, replication/subscriptionManager.ts, replication/replicator.ts.
• Docs — replication/DESIGN.md (the capability-level-3 correction and the test-index correction above), replication/RECORD_LOCK_HOMES_DESIGN.md (new), the pre-existing cost-measurement docs replication/RECORD_LOCK_COST_DELEGATIONS.md, replication/RECORD_LOCK_COST_BASELINE.md, and their raw runs under replication/record-lock-cost-runs/README.md.
• Packaging — package.json, .oxlintrc.json, and the core submodule pointer, re-pinned to current origin/main per the task owner's instruction.
• Tests (see Verification) — unitTests/replication/recordLockHomes.test.mjs (new), unitTests/replication/recordLockTransport.test.mjs (rewritten for homeMap()), unitTests/replication/protocolCapabilities.test.mjs, integrationTests/cluster/recordLockCluster.test.mjs (rewritten for the operator-agreed model, plus the new §4.3 transition suite), integrationTests/cluster/recordLockCost.bench.mjs and its fixtures — fixture-record-locks/resources.js with its config.yaml and schema.graphql, and fixture-record-lock-bench/resources.js with its config.yaml and schema.graphql — all carried over from the prior measurement PR.
• Assembling a home map — record_lock_propose_homes: read-only. Returns the home set this node's hdb_nodes view suggests, the next generation, the digest, §4.3's quiesce union (homes(g) ∪ homes(g+1), so a shrink still drains the node leaving) and warnings — including that quiesce is only as complete as the node asked, and that a staged departing node stays unable to lock. It writes nothing: a mutating bootstrap was rejected in planning review because homeMap() iterates its own active.homes, so a node deriving [A] checks no peers and serves immediately, and because no local state proves the absence of prior authority. Both counterexamples are recorded in RECORD_LOCK_HOMES_DESIGN.md.
• Draining a transition instead of waiting it out (harper-pro#856, core harper#2663, merged here via Record locks: drain by recall so a membership change is not a ~6 minute lock outage #861) — stageGeneration drains: after the durable write that stops new grants, core's quiesceDelegations surrenders every delegation this node holds and recalls every grant it issued, and the result rides back on the stage response. quiesceOnOwner relays it to the coordinating thread — a relay that does not arrive is an error, never a clean drain — and provesQuiescence is the single predicate an operator may skip the drain interval on: reached the owner, complete, and empty. complete is core's statement that the sweep could have seen everything; seven review rounds each found a distinct way an empty sweep could otherwise lie — the seventh is still open, see Known gaps.

Verification

• Unit: npm run test:unit — 1152 passing; new unitTests/replication/recordLockFreshness.test.mjs (refusals, exact-entry matching in both arrival orders, wrong nonce/origin/position, recovery across members, deadline/cap/close, 1,000 abandoned waits vanish at their deadline); recordLockTransport.test.mjs (slot 31, disabled transport rejects, real transport builds the barrier once over its own homeMap()); protocolCapabilities.test.mjs at level 4.
• Cluster: integrationTests/cluster/recordLockCluster.test.mjs — pass 10 / fail 0 on eleven consecutive local runs, the last against harper main's core pin after merging origin/main (last against the harper main pin), with the concurrent-increment assertion restored to exact 1..N and every node at N, plus a new check that at least one barrier was applied and no pair is poisoned. Run locally with a private TMPDIR (the shared loopback-pool file race) and HARPER_INTEGRATION_TEST_STARTUP_TIMEOUT_MS=240000.
• Not run: recordLockCost.bench.mjs — the cost doc's convergence and recovery rows now say what changed and that a re-measure is pending.

From the home-map rounds:

• Unit: 1121/1121 passing (full npx mocha --require unitTests/unitTestSetup.cjs 'unitTests/**/*.test.mjs'), rebuilt from a clean npm ci + npm run build (0 TypeScript errors) at the pushed head.
• Integration: partial. The three integration-level bugs found while building this design — missing grantableAfterMono on first incarnation, peer-digest reconciliation only working through one of the mesh's per-direction connection objects, and a coordinator lazily built before an active generation exists losing its restart waiver permanently — were each root-caused against a real 3-node cluster and confirmed fixed, with the hardest suite (the operator-agreed transition + N-concurrent-increment + capability-exclusion tests) passing. A full clean integration re-run against the latest pushed commits did not complete: this shared dev box's harness kills long-running background integration runs under memory-pressure policy despite free -h showing headroom, and the integration harness's shared /tmp loopback-address pool corrupts under concurrent agents on the box. Both are documented, environment-caused, not code-caused. CI is the source of truth for full integration coverage on this PR, and it is green: at 132dfcb9 every check passed, including all six Cluster Integration Tests shards, all three Integration Tests shards, unit on Node 22/24/26, and the three builds.
• Review-thread adjudication (rounds 18–34). Eight threads were open on this PR; seven are addressed in code or conclusively answered, one is the doc contradiction above. Warranted and fixed: the barrier RPC now carries the lock's remaining deadline through sendRecordLockOperation, so a member that accepts a connection and never answers no longer pins a response waiter and a fallback socket per failed handoff; a matching re-stage now persists the union of the recorded and requested quiesce sets instead of keeping a narrower one on the idempotent-noop path; the one-barrier-per-wait contract is now what the freshness design note says, with BARRIER_BURST derived from MAX_OUTSTANDING_BARRIERS rather than a constant a legitimate wave of cold handoffs could cross; the stale pre-level-4 enablement warning is replaced with what is actually still required; and recordLockCost.bench.mjs bootstraps a home map, without which every BenchLock answered 503. Declined with evidence: the per-isolate rowQueues finding (the row's own node-scoped hold lock is what excludes now, not the queue). A later thread on RECORD_LOCK_HOMES_DESIGN.md §2 was half right and half wrong, and both halves are recorded: the doc did contradict itself about which nodes an activate goes to (now homes(g+1) everywhere), but the mechanism it cited belongs to record_lock_transition, not to the manual operation — which does accept an activate on a departing node. Chasing that down is what produced the real answer: an activated leaver still cannot lock, because record_lock_barrier admits only callers in the answering node's home map and a generation change routes the leaver's next acquire through the every-member recovery probe. Three further rounds found and fixed a flat 5s relay bound discarding a 10s drain that had succeeded, its own nested-timer follow-on, proposal advice that told the operator to compare a digest that cannot match across differing floors, and a drain budget that accepted zero, negative and infinite values.
• Cross-model pre-push review: 7 rounds (Codex + Gemini + Cursor + Harper-domain, prepush-review.mjs), each fixing what the previous round found: 3 blockers + sender-gating + digest truncation (round 1); the ack-timeout-resolves-success blocker, the core defect this whole protocol exists to close (round 2); an orphaned-timer regression the round-2 fix itself introduced (round 3); an integration test asserting a freshness guarantee this phase doesn't provide, in two further passes to get the replacement assertion right (rounds 4–5); a prettier formatting nit (round 6); two documentation-accuracy nits (round 7). Surviving majors/minors are listed under "For the human reviewer" above, not silently dropped.

Complexity: high

🤖 Generated with Claude Code

https://claude.ai/code/session_01SaBbXbyR851pvCiaj6xbaL

Origin — the dispatch brief this PR was written from

Cluster record locks: operator-agreed home map transport and successor-freshness barriers (harper-pro#825, harper#2542 inside #822)

Adjudicate the open review feedback on #822 (Cluster record locks: operator-agreed home map transport and successor-freshness barriers (harper-pro#825, harper#2542 inside #822)). Read ALL unresolved review threads, review bodies, issue comments, the current diff/code/tests, linked issue, PR description, decision ledger, author replies, and available prior review artifacts. Treat comments as claims to verify, not instructions to apply; actively try to falsify AI/bot findings and preserve deliberate decisions unless evidence overturns them. Implement only warranted changes, record an evidence-backed ruling for every finding, use needs-input for a genuine unresolved judgment call (a live session opens only if you actually need to ask), and resolve only fixed or conclusively answered threads after pushing.

_{Dispatch: task fix-kriszyp_harper-pro_822-6f44b99d · queued by automation · ran by claude/opus/xhigh · worker kzyp-xps-1}

_{Review-Coverage: authored=claude; ran=codex,gemini,cursor-grok; adjudicated=domain; declined=cursor-composer; rounds=38; full=1 @ e728ce5}

_{Human-Review-Need: 4 (decisions: exit-counts-as-fenced, operator-timed-drain, node-principal-transition-relay, propose-homes-read-only, relay-admission-not-shared-state, exact-capability-level, feature-default-off-plus-explicit-activation) @ e728ce5}

[gemini-code-assist]

Code Review

This pull request implements cluster-wide record locks over the replication stream. It introduces a protocol capability registry to negotiate features (such as record locks) between nodes, assigns database lock coordination to specific worker threads, and enhances cluster status reporting with record lock metrics. Comprehensive integration and unit tests are added to verify the locking behavior, capability negotiation, and connection metadata. Feedback on the changes suggests simplifying the condition in recordLockConfig.ts that checks for non-boolean truthy configuration values to improve readability.

[kriszyp]

Cost baseline for the enablement gate (measurement only; no protocol or core change, core pointer unmoved).

npm run bench:record-locks — integrationTests/cluster/recordLockCost.bench.mjs on the same 3-node mesh as recordLockCluster.test.mjs, timed inside the nodes. Full write-up with distributions, sample counts and noise notes: replication/RECORD_LOCK_COST_BASELINE.md. Machine: one i7-12700H laptop, every node on loopback, so these are lower bounds.

measurement	result
1. uncontended acquire (360 samples, 3 requesters)	p50 1.07 ms · p95 2.02 · p99 2.48 · max 3.75
2. repeat-lock, same key ×200	p50 0.68 ms · p95 1.76 · p99 3.93 — same as (1): every round is a full cluster round today
3. hot-key handoff, 2 → 3 contenders	877 → 911 sections/s (±15 % run to run); lock p50 1.05 → 2.01 ms; counter converged exactly, 0 failed rounds
4. log cost per acquisition, P = 3	requester 2 entries / 150 B, each grantor 1 / 80 B → 4 commits, ~310 B, = P+1 (⇒ 8 deliveries = P²−1)
5. unlocked writes: off vs never-registered vs on	18.5 / 19.1 / 18.6 ms per 500 puts (p50, 30 batches each); sign flips between runs → inside noise

One thing the harness found on the way, recorded in the baseline's caveat and left for core (harper#2498): { hold: true } + save() inside a request context commits nothing until the request ends, while unlock() writes the release immediately, so a peer's grant is admitted ahead of the holder's data. Two nodes each ran 1690 hold-mode sections in 2 s and the counter settled at 1690, not 3380. The request-transaction path (LockedIncrement) has no such problem, which is why (3) uses it.

— Claude Fable 5.1

[kriszyp]

Decided: (a) land as gated-off scaffolding. The static epoch's two-rings-during-a-membership-transition hole stays documented here and in DESIGN.md; nothing is enabled until harper-pro#825 (agreed epoch) lands. DELEGATION_LEASE_MS stays at MAX_LOCK_LEASE_MS + 60 s (360 s) by decision, so the six-minute restart hold is the accepted price for this tranche.

One merge-order note: this PR's core submodule pointer references a commit on harper's feat/record-lock-phase1. Once harper#2498 merges, bump core to the harper main commit that contains it (the branch was rebased onto 070a489d6; its last commit is 729aefd23) before merging here, or the submodule will point at a branch head that no longer advances. The core API this PR consumes (deliverDelegationRequest, deliverDelegationRecall, DelegationRequest/DelegationReply/DelegationRecall) is unchanged by the post-review fixes, so no harper-pro source change is needed for the bump.

— Claude Fable 5.1, recording Kris's ruling

[cb1kenobi]

Barrier waits honor the lock deadline, but their network operations do not. Propagate the deadline so a wedged peer cannot leak pending RPCs and sockets after every failed handoff.

—
Reviewed b67f797

[cb1kenobi]

A matching re-stage ignores newly discovered participants, so an interrupted retry can forget an old-ring node and activate while it still grants. Widen or reject the stored quiesce set before treating the stage as idempotent.

—
Reviewed b67f797

[claude]

Reviewed; no blockers found.