feat(infra): optional leader-lease endpoint for pooled deploys

AGENTS.md

@@ -144,7 +144,7 @@ Reviewers: treat a hardcoded provider name in code, config, or a committed doc l
 
 ### Deploy invariant: exactly ONE serving API instance (#271)
 
-**Run one serving instance.** Every instance runs `DurableJobWorker`, but its poll and the three recurring sweeps now run **only under a single-leader gate** — a session-scoped Postgres advisory lock (`pg_try_advisory_lock`) on a dedicated, non-pooled connection (**#271**, closed): at most one instance leads, crash recovery is automatic (a dead leader's session releases the lock), and the contract is at-most-one-leader with at-least-once, idempotent handlers — never "exactly once". That guarantee holds on a **session-pinned** Postgres endpoint (a direct connection or a session-pooled proxy); under a **transaction-pooling** proxy (e.g. PgBouncer in transaction mode) the lock can migrate across backends and single-leader is *not* guaranteed — a backend-PID affinity check narrows but does not close the window, so that topology relies on the at-least-once + idempotent contract until a dedicated session-pinned lease endpoint (**#556**) lands. The per-account report concurrency cap (#311) was the last independent in-process blocker, and **#545 closed it**: it now runs on the shared renewable lease (#543), keyed per `AccountId`, each permit pinned to the backend that granted it — so N replicas enforce one combined per-account permit count, degrading to a bounded per-instance ceiling (never fail-open) under a store outage. The IP-keyed auth limiters (#143) were another — and **#544 closed it**: they now run on the shared `IFixedWindowCounter` (#543, Redis-backed with an in-process fallback + alarm), so N replicas enforce one combined per-IP budget instead of N. The step-up grant registry was another — the blocker with teeth — and **#338 closed it**: replay now lives in the shared `IClaimOnceStore` (#543) and logout revocation in the durable per-user `ApplicationUser.StepUpLogoutEpoch`, an integer compared for equality (never a timestamp), so both survive across replicas without a shared clock. **#307 is closed and does not license scaling.**
+**Run one serving instance.** Every instance runs `DurableJobWorker`, but its poll and the three recurring sweeps now run **only under a single-leader gate** — a session-scoped Postgres advisory lock (`pg_try_advisory_lock`) on a dedicated, non-pooled connection (**#271**, closed): at most one instance leads, crash recovery is automatic (a dead leader's session releases the lock), and the contract is at-most-one-leader with at-least-once, idempotent handlers — never "exactly once". That guarantee holds on a **session-pinned** Postgres endpoint (a direct connection or a session-pooled proxy); under a **transaction-pooling** proxy (e.g. PgBouncer in transaction mode) the lock can migrate across backends and single-leader is *not* guaranteed — a backend-PID affinity check narrows but does not close the window, so that topology relies on the at-least-once + idempotent contract, or the operator configures `ConnectionStrings:LeaderLease` to point the lease at a session-pinned endpoint (**#556**, closed). The per-account report concurrency cap (#311) was the last independent in-process blocker, and **#545 closed it**: it now runs on the shared renewable lease (#543), keyed per `AccountId`, each permit pinned to the backend that granted it — so N replicas enforce one combined per-account permit count, degrading to a bounded per-instance ceiling (never fail-open) under a store outage. The IP-keyed auth limiters (#143) were another — and **#544 closed it**: they now run on the shared `IFixedWindowCounter` (#543, Redis-backed with an in-process fallback + alarm), so N replicas enforce one combined per-IP budget instead of N. The step-up grant registry was another — the blocker with teeth — and **#338 closed it**: replay now lives in the shared `IClaimOnceStore` (#543) and logout revocation in the durable per-user `ApplicationUser.StepUpLogoutEpoch`, an integer compared for equality (never a timestamp), so both survive across replicas without a shared clock. **#307 is closed and does not license scaling.**
 
 **Do not extend that list from memory** — it was twice derived wrongly, both times a process-local limiter. Re-derive it by walking every `AddSingleton`/`AddHostedService` under `src/` plus every in-memory state primitive, **plus what each package `Add*`/`With*` extension `Program.cs` calls registers internally** — that last part is not optional and `src/` cannot show it: one innocuous `builder.Services.AddThing()` line can register several services, hosted services included, and the walk will not see one of them (#786). Graph the package at the version `packages.lock.json` resolves and ask it directly — `graphify clone <repo> --branch <tag>`, `graphify update . --no-cluster`, then `graphify explain "<ExtensionName>"` lists what it registers with line numbers. Then exclude deliberately. As of 2026-08-25 that walk finds **19 `AddSingleton` call sites and 1 `AddHostedService`** — but count call SITES, not live registrations: `SharedStateRegistration.cs` registers `IClaimOnceStore` and `IFixedWindowCounter` twice each on mutually exclusive branches (Redis vs the in-process fallback), so at most **17** are live in one process, and **16** when Redis is unconfigured. A bare count here has already gone stale twice; re-run the walk rather than trusting this sentence. The run-then-exit verbs are unaffected: they never start hosted services. The #543 shared-state registrations (`IConnectionMultiplexer` + the `IClaimOnceStore`/`IFixedWindowCounter` ports) are **not** blockers — they are the shared store: #338 wired the `IClaimOnceStore` grant-replay caller and #544 wired the `IFixedWindowCounter` auth limiters (both closed above). #545 wired the report cap onto the lease backends directly: `ReportConcurrencyCapRegistration` constructs `RedisLease`/`InProcessLease` itself and pins each permit to its granting backend, rather than resolving a shared `ILease` port from DI. Redis-backed they are multi-replica-safe, their in-process fallbacks a deliberate alarmed degradation. → [`271-single-serving-instance.md`](docs/decisions/271-single-serving-instance.md)
 
@@ -224,7 +224,7 @@ Two stages, deliberately separate: **CI publishes an image per merge; the releas
 
 **Phase 1.1 (Operational fill) is shipped** — epic #14 closed 2026-08-11: RBAC UI, product catalog / egg-grade management, inventory movement ledger, feed/water/mortality, expenses, payments, dashboard, reports, audit UI, exports, i18n infrastructure. Follow-on work discovered while shipping it moved to epic #15.
 
-**Phase 1.6 (Multi-farm tenancy) is substantially shipped** — epic #530: several farms coexist on one deployment, sign-in takes a farm code, per-account email identity, immediate suspension, operator provisioning (`provision-account`), and all four scale-out blockers closed. Remaining: #357, #388, #537, #556. The decisions and their accepted costs are in [`530-multi-farm-tenancy.md`](docs/decisions/530-multi-farm-tenancy.md).
+**Phase 1.6 (Multi-farm tenancy) is substantially shipped** — epic #530: several farms coexist on one deployment, sign-in takes a farm code, per-account email identity, immediate suspension, operator provisioning (`provision-account`), and all four scale-out blockers closed. Remaining: #357, #388, #537. The decisions and their accepted costs are in [`530-multi-farm-tenancy.md`](docs/decisions/530-multi-farm-tenancy.md).
 
 **Current phase: 1.5** (epic #15, `specs/product/specs.md` §6) — egg product hardening: legacy import, inventory reconciliation, alert center, packaging inventory, additives/supplements, vaccination records, native-speaker es/tl review, deployment readiness, and the Phase 1.1 carryover items on the epic.
 

deploy/.env.example

@@ -21,6 +21,13 @@ POSTGRES_PASSWORD=replace-with-a-strong-password
 # app + migrate services (single host, db never published — plaintext over the private
 # bridge is acceptable there). A real deploy uses TLS/managed Postgres and NEVER sets it.
 # Database__AllowInsecureConnection=false
+#
+# Leader lease (#556). Optional: when your app's Default connection goes through a
+# transaction-pooling proxy (e.g. PgBouncer in transaction mode), point the leader
+# lease at a session-pinned endpoint so the advisory lock stays on one backend.
+# Omit to use ConnectionStrings__Default (correct when the Default endpoint is
+# direct-to-Postgres or session-pooled). Same TLS floor and URI-form support as Default.
+# ConnectionStrings__LeaderLease=Host=db-direct;Port=5432;Database=cluckwork;Username=cluckwork;Password=...
 Jwt__Issuer=cluckwork
 Jwt__Audience=cluckwork-api
 # BOTH keys are checked at boot, and the SERVING process refuses to start unless

src/Cluckwork.Api/Hosting/CluckworkPersistenceServiceCollectionExtensions.cs

@@ -43,9 +43,20 @@ public static CluckworkPersistenceRegistration AddCluckworkPersistence(
                 configuration.GetValue<bool>("Database:AllowInsecureConnection"),
             onWarning: connectionStringWarnings.Add);
 
-        // #271 — the leader lease opens its own dedicated, non-pooled connection
-        // from the same normalised, TLS-floor-validated string the DbContext uses.
-        services.AddSingleton(new LeaderLeaseConnectionString(connectionString));
+        // #271/#556 — the leader lease opens its own dedicated, non-pooled connection.
+        // When ConnectionStrings:LeaderLease is configured, it uses that endpoint
+        // (intended for a session-pinned connection bypassing a transaction pooler);
+        // otherwise it shares the normalised Default string.
+        var rawLeaseConnectionString = configuration.GetConnectionString("LeaderLease");
+        var leaseConnectionString = !string.IsNullOrWhiteSpace(rawLeaseConnectionString)
+            ? PostgresConnectionString.NormalizeAndValidate(
+                rawLeaseConnectionString,
+                isProduction: environment.IsProduction(),
+                allowInsecureConnection:
+                    configuration.GetValue<bool>("Database:AllowInsecureConnection"),
+                onWarning: connectionStringWarnings.Add)
+            : connectionString;
+        services.AddSingleton(new LeaderLeaseConnectionString(leaseConnectionString));
 
         services.AddScoped<TenantStampInterceptor>();
         services.AddDbContext<AppDbContext>((sp, options) =>

src/Cluckwork.AppHost/Program.cs

@@ -29,6 +29,9 @@
     .WithEnvironment("ASPNETCORE_ENVIRONMENT", "Development")
     .WithEnvironment("DOTNET_ENVIRONMENT", "Development")
     .WithHttpEndpoint(name: "http", port: LocalPort("Api"))
+    // #556 — ConnectionStrings:LeaderLease is optional (the lease falls back to
+    // Default). Aspire's direct Postgres is already session-pinned, so no
+    // dedicated lease endpoint is needed here.
     .WithReference(database, connectionName: "Default")
     .WithEnvironment(
         "SharedState__Redis__ConnectionString",

src/Cluckwork.Infrastructure/Jobs/ILeaderLease.cs

@@ -42,9 +42,7 @@ public Task<LeaseStatus> TryAcquireAsync(CancellationToken ct) =>
         Task.FromResult(LeaseStatus.Leader);
 }
 
-// Hands the already-normalised, TLS-floor-validated connection string (registered
-// by AddCluckworkPersistence) to PostgresLeaderLease without a second configuration
-// lookup — the lease opens its own dedicated, non-pooled connection from this exact
-// string. A tiny typed wrapper so DI injects the right string rather than an ambient
-// one.
+// Hands the normalised, TLS-floor-validated connection string to PostgresLeaderLease.
+// Sourced from ConnectionStrings:LeaderLease when configured (#556), otherwise from
+// ConnectionStrings:Default. A typed wrapper so DI injects the right string.
 public sealed record LeaderLeaseConnectionString(string Value);

src/Cluckwork.Infrastructure/Jobs/PostgresLeaderLease.cs

@@ -56,10 +56,10 @@ namespace Cluckwork.Infrastructure.Jobs;
 // single-leader; there is no cheap client-side fix (pinning the backend would mean
 // holding a transaction open for the process lifetime — an idle-in-transaction that
 // starves vacuum, which is worse). The lease REQUIRES a session-pinned endpoint (a
-// direct connection or a session-pooled proxy) for its single-leader guarantee. A
-// dedicated session-pinned lease endpoint that makes pooled deploys single-leader is
-// tracked as a follow-up (#556); on a session-pinned endpoint the affinity check is
-// exact and this whole caveat does not apply.
+// direct connection or a session-pooled proxy) for its single-leader guarantee.
+// ConnectionStrings:LeaderLease (#556) lets an operator point the lease at a
+// session-pinned endpoint; on such an endpoint the affinity check is exact and
+// this whole caveat does not apply.
 //
 // Single-caller: TryAcquireAsync is only ever called from the one worker loop,
 // sequentially, so no internal synchronisation is needed. DisposeAsync runs only

tools/simulation/docker-compose.sim.yml

@@ -85,6 +85,9 @@ services:
       Database__Provider: ${Database__Provider}
       Database__MigrateOnStartup: ${Database__MigrateOnStartup}
       ConnectionStrings__Default: Host=db;Port=5432;Database=${POSTGRES_DB};Username=${POSTGRES_USER};Password=${POSTGRES_PASSWORD}
+      # #556 — ConnectionStrings__LeaderLease is deliberately NOT set: this
+      # stack's db is a direct sidecar (no transaction pooler), so Default is
+      # already session-pinned and the lease shares it.
       # #261/#262 — this service runs Production config, where the TLS floor
       # fails the boot on anything below sslmode=Require (an unset sslmode
       # means Npgsql's 'Prefer', which is exactly the rejected case). The db