[Epic] Improve the Data Protection experience for apps with multiple instances

[amcasey]

The golden path for Data Protection is having a single app instance, ideally on Windows. In that case, pulling anti-forgery, Razor pages, etc into the app, will automatically pull in Data Protection with its default settings. Keys will be stored locally in an XML file on disk and protected with (e.g.) Windows DPAPI. Along the golden path, app developers don't have to think about, let alone configure, Data Protection.

Things get more complicated when there are multiple app instances. For scalability and resilience, it's desirable for any app instance to be able to handle requests from any user session, so each app instance needs to be able to use keys generated by any other instance. Unfortunately, a mechanism for sharing keys is not something Data Protection can figure out on its own, so explicit configuration is required. Still more unfortunately, common approaches like storing the keys in Azure occasionally encounter races and communication failures that result in key-not-found errors.

What are we going to do about it? In 9.0, we're going to focus on:

1. Making existing implementations more resilient against communication errors and initialization races.
2. Evaluating a different approach in which a single, external component is responsible for generating keys and pushing them to shared storage where app instances can share read-only access to them.
3. Documenting best practices for multi-instance apps.

[amcasey]

A list of key-not-found issues: #36157
New pattern issues: #52915, #52916

[josephdecock]

I'm really heartened to see this epic, because in my work on IdentityServer, the by far most common support topic is data protection. The problems that I see are all to do with this:

Along the golden path, app developers don't have to think about, let alone configure, Data Protection.

My users are on the golden path on their local dev machines, but not when they deploy! And IdentityServer uses the data protection services itself to protect sensitive data at rest, some of which can be very long lived (e.g., signing keys, refresh token data, etc), so data protection issues are especially pronounced in that context. I'm really hopeful to see what all comes of this.

[amcasey]

After staring at the code for a while, this seems like a key source of these problems: #21096

Data Protection has a number of catch-and-swallow blocks, but this seems like the only one that wouldn't result in negligible harm (like losing a trace-level log message). Interestingly, a Warning is logged when this happens (which is how I found that issue), so it would be interesting to know if people are seeing this in prod.

[LoggerMessage(12, LogLevel.Warning, "Key {KeyId:B} is ineligible to be the default key because its {MethodName} method failed.", EventName = "KeyIsIneligibleToBeTheDefaultKeyBecauseItsMethodFailed")]
public static partial void KeyIsIneligibleToBeTheDefaultKeyBecauseItsMethodFailed(this ILogger logger, Guid keyId, string methodName, Exception exception);

[amcasey]

Assuming the above is correct, we probably need:

1. better handling for CanCreateAuthenticatedEncryptor failures
2. some sort of retry logic for when the failure is transient

As noted in #21096, the existing recovery mechanism (swallow and eventually generate a new key) is probably helpful in apps with session affinity (not recommended) or a single instance, so we need to try to avoid making things worse for those apps.

[amcasey]

Telemetry: #54451

[ProTip]

Further to josephdecock's point I do not like the implications of point 2 per the description. Namely needing a second, external system to handle key generation and rotation.

This seems like something that could be resolved through distributed locking either in the IXmlRepository or through some distributed locking service provided to DP. ie the EF Core storage already has access to the DB. It could either handle locking itself or configure DP with a db-backed lock provider.

If the external component is the way forward I'd likely implement this as a background timer task that runs on each node and handles the locking.

Distributed locking is so useful and common(and with many good backing options via Redis, Dynamo, K8s, etc) that it would almost make sense to introduce the concept into Asp.Net Core.. But I digress 😄