Record locks: drain by recall so a membership change is not a ~6 minute lock outage

[kriszyp]

A record-lock membership change no longer has to wait out the delegation lease before the next generation can activate. record_lock_stage_generation now drains and reports what, if anything, is still live.

Closes #856. Core half: HarperFast/harper#2663 (kris/856-quiesce-delegations) — merge that first; this PR pins core at it. Branched from feat/record-lock-cluster-transport (#822), so it merges after that.

Why

Staging retracts active in the same durable write that records the staged generation — that is what makes a successful stage be quiescence evidence. But homeMap() needs an active generation, so from that write until record_lock_activate_generation every cluster-scoped lock() on that database answers 503, and §4.3 has the operator wait DELEGATION_LEASE_MS + LOCK_LEASE_SKEW_MS (365 s) in between. Adding one node is a ~6 minute cluster-lock outage for that database, and rendezvous hashing re-homes roughly 1/(n+1) of the keyspace onto a new node, so additions are as exposed as removals.

What changes

stageGeneration calls core's quiesceDelegations after its durable write — the write stops new grants, this drains the existing ones — and returns the result as quiesced. An orchestrator that sees a proven-clean drain on every node in homes(g) ∪ homes(g+1) can activate immediately; anything else falls back to the interval, or record_lock_fence_external, for that node only.

Three things it deliberately does:

• Drains on the coordinating thread. The operations API answers on whichever thread took the request, and coordinator state is per-thread — calling core directly would sweep an empty registry and report "nothing outstanding" while the owner thread held every grant. It relays through the existing record-lock relay, and a relay that does not arrive is an error, never a clean drain.
• Runs outside withRow. The drain waits on live critical sections; holding the database's transition queue for that would block every other stage, fence and activate behind it. The row write has already retracted active, so nothing new can be granted while it runs.
• Never fails the stage. A stage that durably landed must not report failure because the drain did, or the operator retries a transition that already happened. A drain that cannot run is reported as { error }, which reads to an orchestrator exactly like outstanding work.

provesQuiescence() is the single predicate an orchestrator may skip the interval on: reached the coordinating thread, and complete, and an empty outstanding. Every field is checked positively, because the value crosses a worker boundary.

Verification

unitTests/replication/recordLockHomes.test.mjs — 96 record-lock unit tests passing, new ones driving stageGeneration itself: the coordinating thread's result passes through unchanged; a throwing drain becomes { error } without failing a stage that landed; an unwired drain refuses rather than reading as empty; and the proof predicate rejects a malformed reply. Core: 110 passing (HarperFast/harper#2663).

Cross-model pre-push review: six rounds. What they found, in order: the drain ran on the wrong thread; an empty sweep was treated as a proof; ownership continuity, not uptime, is the fact that proof rests on; the first-incarnation waiver cannot stand in for it; retired coordinators' grants outlive their coordinator; and a parameter name plus a non-array guard.

For the human reviewer

• The cost is stated, not hidden: a freshly built coordinator cannot prove quiescence for a full lease, so it reports incomplete. That falls only on a node with no delegations — bootstrapping generation 1, where nothing has been granted and there is no interval to skip.
• This does not narrow the outage to the keys whose home moves. Staging still retracts the whole map. That second lever is noted on Record locks: a membership change is a ~6 minute cluster-lock outage; drain by recall instead of waiting out the delegation lease #856 as the follow-on; with this one in place the window is a live critical section rather than a fixed interval.

🤖 Generated with Claude Code

[claude]

Missing test: quiesceOnOwner's cross-thread relay path — the actual "drain by recall" mechanism this PR adds — has no coverage at all.

quiesceOnOwner is the new public entry point this PR wires in as the drain reader (recordLockTransport.ts: setHomesDrainReader((database, deadlineMs) => quiesceOnOwner(database, deadlineMs));). It has two branches that matter: answering locally when ownership.ownsDatabase(database), and relaying to the owner worker via the new 'quiesce' RelayKind otherwise (with fail-closed handling when the relay times out, there's no owner, or the reply isn't a proper { outstanding: [...] } shape).

The only tests added in this PR (unitTests/replication/recordLockHomes.test.mjs) exercise stageGeneration/drainForStage/provesQuiescence entirely through setHomesDrainReader(mock), which replaces quiesceOnOwner wholesale — none of them call quiesceOnOwner itself. unitTests/replication/recordLockRpc.test.mjs doesn't exist, and integrationTests/cluster/recordLockCluster.test.mjs (unchanged by this PR) has no reference to quiesce/outstanding/provesQuiescence anywhere.

So the code that this PR's own title is about — reaching the actual coordinating worker for a database during a membership change, across the new 'quiesce' relay hop — is exercised by nothing: not the happy path (relay reaches the owner and gets a real QuiesceResult), not "no owner yet," not a relay timeout, not a malformed reply. Given the sibling function provesQuiescence needed a follow-up fix in this same PR for a shape it didn't handle correctly, this relay path (relay()/executeLocally()/routeFromMain() for kind: 'quiesce') is exactly the kind of new branch that benefits from a direct test — e.g. a unit test with two fake worker "threads" (mirroring the existing recordLockTransport.test.mjs style) verifying quiesceOnOwner returns the owner's real result when relayed, and throws the fail-closed 503 when the relay times out or no owner is registered.

[claude]

Reviewed the actual diff for this PR (cb2340d..de3f39a: recordLockHomes.ts, recordLockRpc.ts, recordLockTransport.ts wiring, and the core submodule bump) — the drain-by-recall mechanism itself is well-built and the PR's own last commit already fixed the one correctness gap a reviewer would look for (provesQuiescence checking every field positively). One blocker: quiesceOnOwner's cross-thread relay path, the actual mechanism this PR adds, has no test coverage anywhere (unit or integration) — see inline comment on replication/recordLockRpc.ts.