Record locks: one operator call to apply a home map across the cluster, given an explicit node list #862

Summary

Applying a record-lock home map is an N×2 loop today: record_lock_stage_generation on every node in homes(g) ∪ homes(g+1), then record_lock_activate_generation on every node in homes(g+1), with the operator carrying one identical homes[] list to each and deciding when the drain is done. record_lock_propose_homes (harper-pro#822) removed assembling the list by hand; it did not remove the loop.

This issue adds the single operator-facing call that drives the whole §4.3 transition, given an explicitly supplied list of expected nodes.

Why the explicit list is what makes this safe

The rejected design in harper-pro#822 had a node derive the ring from hdb_nodes. That is unsafe for a specific, mechanical reason worth restating, because it is easy to re-propose: homeMap() iterates its own active.homes, so a node whose derived set is [A] checks no peers at all and serves its ring immediately. Two nodes with disjoint or merely incomplete views each get a usable map and both arbitrate the same key. A digest cannot detect a participant omitted from the set being digested.

When the operator states the set, there is nothing to omit. The list is complete by construction, so "ask every named node, fail if any does not answer" becomes a complete check rather than a best-effort one, and unreachable becomes a hard failure — which was not even expressible before. This operation may therefore orchestrate; it still may not derive.

Scope

A new requiresSuperUser operation on replication/recordLockHomes.ts, e.g. record_lock_apply_homes { database, homes[], generation? }:

1. Survey. Ask every node in homes (and every node any of them reports in its current active/staged ring) for its current home-map state. Refuse unless every named node answers. Refuse if any node reports a ring containing a node the operator did not list — that is "you forgot a node that is still granting", and it is the residue of the second objection in Cluster record locks: operator-agreed home map transport and successor-freshness barriers (harper-pro#825, harper#2542 inside #822) #822 (no local state proves the absence of prior authority).
2. Cross-check digests. digestOf(generation, canonicalHomes) is deterministic over the canonicalized set; refuse unless what the nodes report is identical-or-absent.
3. Stage everywhere, in homes(g) ∪ homes(g+1), with the one canonical list.
4. Use the drain. Each stage returns quiesced (harper-pro#856 / harper#2663). If provesQuiescence() holds for every node, activate immediately — no DELEGATION_LEASE_MS + LOCK_LEASE_SKEW_MS wait. Otherwise do not activate: return the per-node results and the earliest time activation would be safe, for a second call.
5. Activate everywhere in homes(g+1). A departing node is staged and deliberately never activated.
6. Report per node, never a boolean.

The two things that need designing, not just coding

Auth for the hop. sendOperationToNode connects with mTLS, so a peer authenticates it as a node principal, not super_user — and stage/activate are requiresSuperUser. Do not forward the operator's credentials per hop. Add a peer-callable operation authorized by the node principal, exactly as record_lock_delegate / record_lock_recall / record_lock_barrier already are (recordLockRpc.ts's principalNodeName), whose handler re-validates locally: the receiving node checks the generation, the digest and its own row before writing. The node principal then relays a proposal each node independently verifies, rather than authorizing policy.

Partial failure, which is why this is feasible at all. stage retracts active in the same write, so a half-staged cluster has some nodes quiesced (their locks 503) and none double-granting — the failure direction is toward refusal. planStage and planActivate both have explicit idempotent noop branches for the same generation+digest, so the operator retries the same call. Specify the retry contract and make the response say exactly which nodes are in which state.

Acceptance

• One call applies generation 1 to a fresh three-node cluster, and a cluster integration test asserts locks work afterwards with no manual per-node loop.
• One call performs a topology change, activating immediately when every node proves quiescence, and asserting the lock outage is bounded by the drain rather than by DELEGATION_LEASE_MS.
• A node that is unreachable, disagrees on the digest, or reports a ring member the operator did not list causes a refusal before anything is staged.
• A failure injected between stage and activate leaves the cluster in a safe, documented state, and re-running the same call completes the transition.
• The peer-callable hop rejects a caller that is not a node principal, and re-validates rather than trusting the payload.

Not in scope

Deriving the list (record_lock_propose_homes remains the read-only helper). Narrowing the outage to only the keys whose home moves — the remaining lever on harper-pro#856.

Dependencies

Branch from feat/record-lock-cluster-transport (harper-pro#822) — none of this exists on main. It also needs the drain from harper-pro#861, which is already merged into that branch.