Background worker has no single-runner guarantee — double-runs if scaled >1 instance

[mforce]

Coding brief — read first. Start with epic #530 (the canonical decision record); this is slice S1 of that epic. This body is complete and current as of 2026-08-16, folding every review correction — no comments to cross-reference. Repo conventions in AGENTS.md apply. Definition of done = the acceptance criteria at the end. Originally filed under the #244 deployment-readiness audit; now a Phase 1.6 scale-out blocker.

Problem

The background worker has no single-runner guarantee, so it silently double-runs if the app is ever scaled past one instance. Every API instance registers DurableJobWorker as a hosted service; the poll selects Pending jobs with no row claim — no FOR UPDATE SKIP LOCKED, no lease (DurableJobWorker.cs) — and the three recurring sweeps (DailyEntryLockSweep, RefreshTokenPurgeSweep, IdempotencyRecordPurgeSweep) ride the same poll and run unconditionally per instance. With one instance this is fine; two instances each process the same jobs and both sweep.

The contract is "at most one active leader", NOT "exactly once"

This is the correction from the epic #530 review, and it is load-bearing — do not write "exactly once" into the finish line. A Postgres advisory lock provides mutual exclusion only: a session lock is released when the session dies, so a leader can lose the lock while its work is still running, and a process can crash after an external effect but before marking a job complete. Exactly-once is therefore unachievable at this layer. The achievable, correct contract:

• at most one active leader under ordinary two-replica operation;
• durable jobs are at-least-once; handlers are idempotent (this is the property that makes at-least-once safe);
• recurring schedules have durable next_run/claim state, not "whoever holds the lock right now may run";
• jobs claimed with leases/fencing or FOR UPDATE SKIP LOCKED, with crash recovery.

Scope

Move the worker and the three sweeps behind a Postgres advisory-lock lease (the lease is deliberately Postgres, not one of #543's Redis ports — a Redis lock's failure mode is precisely the double-execution the lease prevents; epic decision 13). Add crash recovery so a dead leader's lease is reacquirable.

Note: the two purge sweeps are already idempotent deletes; DailyEntryLockSweep reads-then-writes under an optimistic Version token, so a losing replica currently throws a concurrency exception its per-account catch logs as a DB fault. The lease removes the double-run rather than relying on that catch.

Sequencing

Slice S1 of epic #530, after #543 (S0). The lease mechanism itself does not consume #543's ports, but the scale-out slices land in order and the epic gates a second replica on all four (#271, #338, #544, #545) closing.

Acceptance criteria

• At most one active leader runs the worker loop and the three sweeps under two instances.
• A two-instance integration test proves each job and each sweep runs once per due interval, not once per instance (precedent: MultiInstanceIdempotencyTests).
• Tests cover leader connection loss mid-work, process death after an effect but before completion, and lease reacquisition by a survivor.
• Durable jobs are at-least-once with idempotent handlers; the finish line does not claim "exactly once".
• Closing this alone is not presented as permission to scale — Move the step-up grant registry to a shared store before running >1 replica (follow-up to #308) #338, Distributed IP-keyed auth rate limiters with in-process fallback #544 and Distributed per-account report concurrency cap with a local ceiling fallback #545 must also close (recorded by Docs: multi-farm tenancy ADR + AGENTS.md / GLOSSARY sync #537 in AGENTS.md).
• The concrete replica cap and the later scale change are coordinated in the deploy tracker, not here.

Verify

dotnet test Cluckwork.sln

[mforce]

Cross-link: epic #530 (Phase 1.6 — Multi-farm tenancy).

This issue is not caused by #530 and is not a slice of it — the epic keeps the one-serving-instance invariant exactly as AGENTS.md states it, and closing nothing here.

What changes is the cost of living with it. Today a single serving instance serves one farm, so the ceiling is one farm's load. After #530 that same single instance serves every farm on the deployment, and this issue is one of the four independent blockers that must close before a second replica is allowed. Multi-farm therefore raises the priority of this work without altering its content.

Recorded here so the connection is not re-derived later.