Identity provider key caching behavior causing lookup conflict

[coolgang123]

Problem:
UAA caches the key from Identity provider. The default cache eviction time is 10 minutes. When the Identity provider changes the key, the UAA may end up with stale key in the cache. All the token validations will fail during stale key period.

Resolution:
In case of signature validation failure, invalidate the cache and retrieve the latest/current key from Identity Provider and use it for validation. This behaviour is applicable for all signature validation failure.

Another potential solution considered:
Provide an endpoint or update existing endpoint that enables Identity provider to trigger cache invalidation upon key change.

Testing done:
Unit test and IT added
Manual testing done by deploying a UAA and an Idenity provider.

[linux-foundation-easycla]

• ❌ The email address for the commit (bc0eec3) is not linked to the GitHub account, preventing the EasyCLA check. Consult this Help Article and GitHub Help to resolve. (To view the commit's email address, add .patch at the end of this PR page's URL.) For further assistance with EasyCLA, please submit a support request ticket.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/187737226

The labels on this github issue will be updated when the story is started.

[peterhaochen47]

line 896: It took me a while to understand what happens here. Is this true: in this line, the code is going to fetch the first key (which does not match xCodeToken), and then immediately re-fetches the key (getting the second key) and retries (succeed). If so, the //with first key comment is a little confusing, since it's eventually not using the first key anymore.

[peterhaochen47]

line 897: And then here it's gonna use the previously cached key (aka the second key), and it works.

[peterhaochen47]

Is this error message something the code consumes internally, or is it returned to the user? If the former then it's fine, if the latter we then might need to look deeper to see if there's any unexpected effects (since the user interface has technically changed).

[strehle]

Please explain the conflict situation in more detail, because we use the current mechanism for a longer period and don't have problems with rotation. The assumption for rotation is:

1. that the external IdP does a key change with a unique key id (kid) in JWKS_URI
2. that the external IdP keeps non active keys for a period of time (e.g. a Day) in JWKS

The cache is currently hard coded in amount of entries and 10 min, yes. So we maybe should have options to configured these parameters, but from our point of view we would increase evict time in cache from 10 min to 24 hours.

If you have an IdP which does not support a key rotation in the described way, then your option to mitigate this is, that you configure (and later update) the IdP configuration with a tokenKey , e.g. https://docs.cloudfoundry.org/api/uaa/version/77.10.0/index.html#oauth-oidc -> search for config.tokenKey.
With that, you are not using the cache, but if my assumptions with your IdP are correct, then in your case you should not use a cache but the configured signature validation key. You can control this key.

Allowed structure of this tokenKey is either

• JSON JWKS -> body of https://login.uaa-acceptance.cf-app.com/token_keys
• PEM public key -> https://www.baeldung.com/java-read-pem-file-keys#2-get-public-key-from-pem-string
• PEM x509 certificate -> similar to key but PEM header is -----BEGIN CERTIFICATE----- ...

[coolgang123]

The following is the reason for proposed fix:

We have a customer who was running UAA version 74. The UAA used to issue a GET /jwks call with every token request for grant_type of urn:ietf:params:oauth:grant-type:jwt-bearer.

Recently, they have upgraded to UAA versions 77 which do NOT issue a callback on every request and appears to be caching the keys for some length of time. As stated, the cache duration is 10 minutes.

Customer refreshes keys in agressive manner. They generate new jwks keys on startup and every 10 minutes thereafter. So, the key generation runs into a race condition with caching and thus they encounter could not verify token signature errors.

The rotation assumptions may not work in the this aggressive rotation scenario.

[strehle]

The following is the reason for proposed fix:

We have a customer who was running UAA version 74. The UAA used to issue a GET /jwks call with every token request for grant_type of urn:ietf:params:oauth:grant-type:jwt-bearer.

Recently, they have upgraded to UAA versions 77 which do NOT issue a callback on every request and appears to be caching the keys for some length of time. As stated, the cache duration is 10 minutes.

Customer refreshes keys in agressive manner. They generate new jwks keys on startup and every 10 minutes thereafter. So, the key generation runs into a race condition with caching and thus they encounter could not verify token signature errors.

The rotation assumptions may not work in the this aggressive rotation scenario.

Okay, I now get it. So I assume you run on UAA before https://github.com/cloudfoundry/uaa/releases/tag/v74.30.0 ?

Because with this version we introduced the cache, but in your case I simply would answer, you should deactivate the cache.

Unfortunately you dont have the option to do this, so I think a small fix to allow to omit keyCache would solve your problem.

btw: For OIDC IdPs we have 2 caches

1. well-known -> if you have defined discoveryUrl . Do you use this ?
2. tokenKey -> if you either have defined discoveryUrl only or tokenUrl with tokenKey

I would start to omit keyCache only because I assume your IdP endpoint wont change, correct ?

Adding a option cacheJwk=true would be an easy task. You could set this then to false and would have the same logic in 77.11.0 as with 74.5.xxxx . What do you think ?

[strehle]

see #2920

[strehle]

think we can close this because pr #2920 is merged