DataProtection: IDataProtector.UnProtect can force refresh the key ring if it fails to unprotect

[kevinlo]

I have an App1 that will create the cookie and have 20 instances of App2 and 20 instances of App3 that will read the cookie. The data protection key is stored in Couchbase.

When these applications are first deployed without the key in the Couchbase, all 20 instances of App2 and 20 instances of App3 are started. They all CreateNewKey. Because the Couchbase indexes are updated cyclicly, every 20ms, the GetAllKeys call may not see the key added by the CreateNewKey immediately. I finally see 62 keys created.

I think because of that 20ms time lag, the GetAllKeys call in some instances of App2 and/or App3 get more keys than others. The App1 is started last and it will get all the keys and so its default key is newer than some instances of App2 and App3. Therefore, the cookie protected by the newest default key in the App1 cannot be unprotected by some instances of App2 and App3.

Currently, KeyRingBasedDataProtector.UnprotectCore calls the _keyRingProvider.GetCurrentKeyRing to get the CacheableKeyRing. The CacheableKeyRing.IsValid just checks if the keys in the key ring is not revoked and expired. In the above case, the keys are all valid and the CacheableKeyRing will be used, and it does not have the latest key from the DB.

Should the IKeyRingProvider.GetCurrentKeyRing have a forceRefresh parameter default to false so that if the KeyRingBasedDataProtector.UnprotectCore fails, it will call GetCurrentKeyRing(true) to force it to refresh the key rings and try again OR provide another way to tell the IKeyRingProvider to discard the cache? KeyRingBasedDataProtector.UnprotectCore can try to unprotect again with the refreshed key ring. At least it can force it to discard the cache and get the refresh one so next unprotect call would be good.

The current workaround I think of is making the App1 startup first so it will generated the key. When the App2 and App3 are started, they can read that key and App2 and App3 always have the same or newer key as App1.

I think that application dependency is a possible solution but it would be nice if the ASP.NET Core Data Protection can handle this situation magically so I don't need to care about the application dependency.

[blowdart]

We don't refresh like this because it's easy to send a fake payload with a new key ID in every request, which would then block all incoming requests until the refresh finishes. It's too easy a DoS vector.

Your workaround really is the only solution. You could, in startup, just manually spin up a protector and call protect so you're not waiting for the first inbound call to start creating a keyring.

[kevinlo]

The DataProtectionStartupFilter.Config function will call the IKeyRingProvider.GetCurrentKeyRing to preload the ring, I don't need to put anything extra in the Startup. The problem is IIS does not start the process until it has the request. IIS8 or later has the preloadEnabled attribute that I can try.

My current workaround works only because the protect\unprotect is one way - always have the App1 protect the cookie and App2 and App3 to unprotect it. If the data protection for the shared apps is in any direction, it will still have the problem. Is there a better solution to make all shared apps having the same keyring?

[natemcmaster]

We're seeing issues related to this pop up and are going to investigate possible solutions. I've assigned myself and put it in a 3.0 milestone so this is tracked and reviewed in the coming weeks.

[kevinlo]

I just come across another problem during a few days of performance testing. As said in the above thread, we have App1 to create a cookie and App2 and App3 will use the same data protection key to read the cookie. App2 and App3 are started and ended per user session, During the few days of performance, each day we have a few hours that can see the performance hit. I find out that, when some of the App2 or App3 started, they had transient problem in getting the data protection keys from the Couchbase and the IXmlRepository.GetAllElements fails. Therefore, a new key is created and the StoreElement is called and the new key is saved to the Couchbase. We have 3 nodes before the load balancer. After around 18-24 hrs (KeyRingRefreshPeriod), the App1 in one of the node has the GetAllElements call, it will now have that new key set as Default key. The cookie created by the App1 in that node cannot be decrypted by any App2 and App3 while those App1 in the other two nodes are still ok. However, the App1 in the other two nodes will have the GetAllElements call one by one and they are used the new key. Then, App2 and App3 will have the GetAllElements call and then they can use the new key to decrypt. Therefore, from the first App1 in a node getting the new key till all App2 and App3 in all nodes getting that new key, we have seen performance degradation.

One workaround now is changing App2 and App3 to DisableAutomaticKeyGeneration so only App1 can generate new key. But then after the 90 days key expiration, App1 in one node will still generate a new key and we still need to make sure App1 in other nodes and App2 and App3 get that new key.

What is the best practice to achieve this key synchronization among different apps and in different nodes?

[natemcmaster]

I haven't had time to make progress on this one. @blowdart @Pilchie who can we hand this off to while I'm on leave? This one needs some design and further investigation.

[Pilchie]

@HaoK - is this something you can take a look at while @natemcmaster is on leave?

[HaoK]

Sure I will take a look