Data Protection has no persisted key ring, so Identity's token providers break across restarts and replicas

[mforce]

Important

Amended. The failure boundary below is overstated. An adversarial review of the #788 design
corrected it. ASP.NET Core Data Protection persists keys by default in an environment-dependent
location, and even an ephemeral ring survives across requests within one process. So "a second
replica breaks it" is not automatically true, and src/Cluckwork.Api/Dockerfile already
acknowledges default keys under the application user's home.

Still accurate and worth fixing. There is no explicit, durable, shared configuration. The
real risks are (a) container replacement losing an unmounted key ring, and (b) one instance being
unable to consume a token another instance protected, when no shared ring exists. Generation and
consumption are currently adjacent calls in the same operation, which is why nothing fails today.

Read the body below with that correction applied.

---

Found while evaluating what ASP.NET Core Identity offers for #788. This is independent of that work and is a latent defect today.

The gap

The repo has no Data Protection configuration anywhere. I checked for each of these and found none: AddDataProtection(), PersistKeysToDbContext, PersistKeysToFileSystem, IDataProtectionProvider/IDataProtector usage, and key-ring settings in any appsettings*.json.

Meanwhile, src/Cluckwork.Api/Hosting/CluckworkIdentityServiceCollectionExtensions.cs does register AddIdentityCore<ApplicationUser>()...AddDefaultTokenProviders(). AddDefaultTokenProviders() registers DataProtectorTokenProvider<TUser>, which uses IDataProtector. With no explicit configuration, ASP.NET Core falls back to a local, unpersisted key ring. That ring is ephemeral in a container, and it is per-process rather than shared.

Why it is latent rather than currently broken

IdentityProvider calls UserManager.GeneratePasswordResetTokenAsync today, so the app does exercise the data-protector path. It does not visibly break because ResetPasswordAsync consumes the generated token in the same request, in the Owner-driven "set a user's password" flow. The token never has to survive a restart, and a different process never has to validate it.

So this is a defect that needs a second condition to appear, not an outage. Two things would trigger it:

1. A real forgot-password flow. The app emails a token and redeems it minutes or hours later, possibly on a different replica. Tokens would fail validation after any restart or on any other instance.
2. A second replica. The repo is openly building toward this. Background worker has no single-runner guarantee — double-runs if scaled >1 instance #271 closed its scale-out blockers, Shared-state ports: Redis implementations + in-process fallback #543 added shared state, and Leader lease: dedicated session-pinned endpoint for transaction-pooled deploys #556 tracks the remaining lease work. Instance B cannot verify a data-protector token that instance A minted.

Why it matters beyond password reset

Anything using Identity's in-box token providers inherits this:

• TOTP enrolment (Add TOTP and WebAuthn/passkey as step-up factors (follow-up to #308) #320), covering the authenticator key and recovery codes. The verified finding there is that TOTP is otherwise nearly free in .NET 10, so a missing key ring is the main real prerequisite for that slice.
• Email confirmation, if ever added.
• Any future use of GenerateUserTokenAsync/VerifyUserTokenAsync.

It does not affect the OAuth work in #788 as designed. That design chose reference tokens, which are database lookups rather than protected payloads. OpenIddict does use Data Protection for some internal state, so fixing this first is still the safer order.

Fix

Configure a persisted, shared key ring. Two viable shapes:

• PersistKeysToDbContext<AppDbContext>() stores keys in Postgres, which every replica already shares and which is already backed up. It adds one table, so it needs one migration (feat(eggs): make cracked and dirty eggs sellable stock via condition grades (#396) #407-compatible) and a regenerated docs/schema/ (chore(schema): generate PostgreSQL schema documentation #417). This is the option I would default to. It adds no new infrastructure dependency, and operations already look after the storage.
• Redis, using the instance Shared-state ports: Redis implementations + in-process fallback #543 already provisions. It fits the shared-state work, but it makes Redis an auth-critical dependency. Shared-state ports: Redis implementations + in-process fallback #543's in-process fallback would also be wrong here, because a silently degraded key ring is worse than a missing one.

Either way, keys should be encrypted at rest. On a host-agnostic deployment, that means a certificate or key supplied through config rather than a provider-specific KMS. This follows the repo's host-agnostic rule.

Not claimed

No user-visible bug exists today, and nothing is currently insecure. With an unpersisted key ring, tokens fail closed (validation fails), not open. The claim is narrower. A mechanism the app already registers cannot work across the restart or replica boundary that the project is deliberately moving toward, and two planned features (#320 and a real forgot-password flow) would expose it immediately.

[mforce]

Sequencing. This is the first thing to build. It is a prerequisite for #795 (stand up OpenIddict), and it is independent of all the MCP and OAuth design work. It is also small and self-contained, which makes it the cheapest test of whether that design works in real code.

Read the amendment banner at the top before starting. The original framing of the failure boundary was overstated, and the banner corrects it.

[mforce]

This matters beyond TOTP. OpenIddict may depend on it too.

#795's OAuth server can use the ASP.NET Core Data Protection token format instead of JWT. Per OpenIddict's docs, that format "relies on their own key ring, distinct from the credentials" used for certificates. If #795 takes that route, the Data Protection key ring stops being only a TOTP prerequisite, and OAuth token protection starts to depend on it.

That raises the stakes on the second risk in the body above, where one instance cannot consume what another protected. For password-reset tokens, that is a latent, narrow problem. For OAuth tokens across replicas, it would be a live one.

This does not change what needs building here. It changes how much depends on it. It is worth knowing before choosing between the two options priced on #795.

[mforce]

Moved from milestone "MCP server support" to "OAuth 2.1 authorization server" in the 2026-10-06 re-evaluation of #789. This is OAuth-side work. #794 is #795's prerequisite, and #793 concerns the SPA login. No MCP slice depends on it directly.

[mforce]

Amendment, 2026-10-06: what #514 changed for this slice

The body and comments above stay as written. Paths were checked against main at 8cb14e67. Nothing has configured Data Protection since this issue was filed. AddDataProtection and PersistKeys* still appear nowhere under src/.

If this slice takes the recommended Postgres option, the new key table meets four guards that did not exist, or did not apply, when the body was written.

• Table owner. The key table's entity type, DataProtectionKey, lives in a Microsoft namespace. TableOwnerRealModelTests fails until a row in TableOwnerOverrides (tests/Cluckwork.Application.Tests/Architecture/RealModuleLedger.Tables.cs) names its owner and reason. The AspNet* Identity tables go to Access, while durable_jobs and idempotency_records go to Platform. The key ring serves Identity's token providers and, later, OpenIddict, so Platform looks like the better fit. The author decides and records the reason in the row.
• Tenant-bypass floor. The table has no AccountId. TenantBypassDiscoveryTests.DiscoveredSurface_Floor asserts exact set equality over filter-free entities, so it goes red until someone adds the entity with a stated reason. Do not relax the assertion to a subset check.
• Record chronology (Standardize business-record timestamps and chronological list ordering #819). Every mapped type must be classified as a created record, a mutable record, or an explicit framework or infrastructure exclusion. Add DataProtectionKey to the exclusion list in src/Cluckwork.Infrastructure/Persistence/PlatformBusinessRecords.cs, beside the Identity types.
• Placement. PersistKeysToDbContext<AppDbContext>() requires AppDbContext to implement IDataProtectionKeyContext. Persistence edits follow src/Cluckwork.Infrastructure/Persistence/AGENTS.md. Register the key ring with the Platform services in src/Cluckwork.Api/Hosting/Modules/CluckworkPlatformServiceCollectionExtensions.cs, not inside AddCluckworkIdentity, so it does not belong to Access alone.

Two earlier rules still apply and are easy to miss:

• The migration and regenerated docs/schema/ ship in the same PR (feat(eggs): make cracked and dirty eggs sellable stock via condition grades (#396) #407, chore(schema): generate PostgreSQL schema documentation #417).
• The certificate for encryption at rest is a new required config key. Add it to the sim harness (tools/simulation/bootstrap.sh, docker-compose.sim.yml, verify-harness.sh, Sim harness (#243) rotted silently: 4 breakages, no CI ever ran it #370) and to src/Cluckwork.AppHost/Program.cs (Developer experience: add an Aspire AppHost for local orchestration and observability #565) in the same PR. Neither runs in CI.