JWKS with new kids gets not loaded after a change of JWKS contents

[hofmannmich]

We rotate our JWKS entry from time to time. On dev-stage we rotate every 4h (just for testing purposes). Traefik with this plugin was installed static (not loading during startup). And the configuration of JWKS is set to load on startup (default value).

After rotating the JWKs in JWKS with changed kid, the plugin does not reload the new JWK from the endpoint. I added the output of the changed JWKS response during time change.

new contents:
dev-10.03.2025-13:26:00.output.json

old contents:
dev-10.03.2025-11:23:52.output.json

remark: some of the JWK in the two output files stays the same (also only for testing purposes). Only one JWKS gets inserted (Eb7byZME1IbSn9MZay8_04D5s-02i68TGxwk_lEyO0A) and one gets rotated out (5dqj7bThbhAW1HF7GDGl5OBKOfUon5Q1umS5tD5WbfQ).

After changed JWKS contents on our IDP: every request gets blocked with 401 by the plugin. And I can not see a reload of the new kids in the log as it was done in the startup-sequence of treafik.

After restarting treafik the new contents of JWKS will be fetched and printed to log. And the request will be allowed by the plugin (no more 401).

[agilezebra]

Thanks Michael. The test key rotation in jwt_test.go should be explicitly testing for this exact scenario but it's possible that something is different about yours. I'm going to recreate your scenario with the 4 keys and see if I can get it be misbehave here. I'll also see what logging is showing about attempts to refresh the keys from the IDP when a new kid is seen, and then add any additional logging if it's needed to help you diagnose on your end. Please can you email me your traefik log (preferably just enough to show the issue without any other activity if you can recreate it easily).

[agilezebra]

When I run that test in isolation, this is the output:

jwt-middleware% go test -run TestServeHTTP/key_rotation                                                                   master
fetched openid-configuration from url:http://127.0.0.1:52069/.well-known/openid-configuration
fetched openid-configuration from url:http://127.0.0.1:52069/.well-known/openid-configuration
fetched key:uqN80F-q72GH7Fg-aRhEK9ec28iFuKk_YgQYaNOfyCQ from url:http://127.0.0.1:52069/.well-known/jwks.json
fetched openid-configuration from url:http://127.0.0.1:52069/.well-known/openid-configuration
fetched key:stk1r-OBsLHJXRgfrkts9NzyjD-IJqRjStp-rZmoljc from url:http://127.0.0.1:52069/.well-known/jwks.json
key:uqN80F-q72GH7Fg-aRhEK9ec28iFuKk_YgQYaNOfyCQ dropped

you should see something similar in your logs when you have a new kid

[hofmannmich]

I can not see your log output when the kid has changed. This was my first guess to look for something like this. I only see the first fetch of all keys on startup (in ERR log level). But then, nothing more...
my config:

        secure-x:
          plugin:
            jwt:
              issuers: [http://myhost/womo]
              insecureSkipVerify: [myhost]
        secure-y:
          plugin:
            jwt:
              issuers: [http://myhost/womo]
              insecureSkipVerify: [myhost]

Because I need two types of JWT validation (based on claims in JWT).

and in static config:

    experimental:
      localPlugins:
        jwt:
          moduleName: github.com/agilezebra/jwt-middleware  

Maybe the two configs make a problem?

And here are my log-configs:

    log:
      level: INFO
      format: common
    accessLog:
      format: common
      fields:
        names:
          StartUTC: drop

[agilezebra]

The two configs shouldn't make any difference. I usually use two configs: one for interactive web and one for api only.
I've created a similar scenario to yours and everything is working as expected. I can test with a new key and I will see something like key auB10cYNWdbII6KooYWea2DAdVUD1AIy26jEdbCS_2Y: fetched and no match" module=github.com/agilezebra/jwt-middleware plugin=plugin-jwt if testing with the key auB10cYNWdbII6KooYWea2DAdVUD1AIy26jEdbCS_2Y before it is rotated in and
fetched key:auB10cYNWdbII6KooYWea2DAdVUD1AIy26jEdbCS_2Y from url:https://yourhost.com/.well-known/jwks.json" plugin=plugin-jwt module=github.com/agilezebra/jwt-middleware after it has been rotated in.

The fact that there are no messages at all about fetching keys makes me a little suspicious that there isn't an iss claim in the jwt matching your IDP. Can you confirm this. Without an iss claim, it would not know where to go to refresh the keys and would silently disallow any unknown requests.

[hofmannmich]

We have https for calls from the internet and http for calls inside OpenShift. The hostname <my-idp> can only be resolved from the internet and <my-idp-internal-hostname> is resolvable from inside OpenShift.

The iss claim in our JWT points to: https://<my-idp> and the JWKS can be reached via https://<my-idp>/jwks
BUT: the config-endpoint is: https://<my-idp>/<realm>/.well-known/openid-configuration

For cluster-internal communication:
The endpoint in the plugin config is set to: issuers: [http://<my-idp-internal-hostname>/<realm> and here you can reach http://<my-idp-internal-hostname>/<realm>/.well-known/openid-configuration which shows you that JWKS is here: http://<my-idp-internal-hostname>/jwks

For me it looks like you load the initial config from issuers config http://<my-idp-internal-hostname>/<realm>/.well-known/openid-configuration which points to http://<my-idp-internal-hostname>/jwks

And for the first time kid is new you reload JWK values based on the iss claim.

If I am correct: you have 2 ways to fetch JWKS values.

Is it possible to reload new kid entries based on the issuers config? I think this makes your plugin more secure, because you trust only configured issuers and not issuers from the iss claim. If you reload from the iss claim, then I can present your plugin an issuer which has not been configured and you load the JWKS from the new idp.

Setting the iss claim to https://<my-idp>/<realm> does not solve the problem, because your plugin can not call the https endpoint from inside OpenShift.

[agilezebra]

There is actually only one way to fetch JWKS values and that's from the issuers entries (ignoring static keys specified in config). It might seem like two because it's triggered in two ways:
1 It will preload all of the keys from all of the issuers on startup (unless it's told not to with skipPrefetch).
2) Regardless of preloading, when it sees an unknown kid, it looks to refresh keys from the known issuer corresponding to the iss value in the JWT. This was a deliberate design choice because it would be inefficient and probably poor behaviour for us to hit up all the IDP endpoints whenever we see an unknown kid.
I think it's a fairly unusual scenario for {iss}/.well-known/openid-configuration not to be reachable or for iss not to be in your list of issuers but I fully appreciate that there are all sort of complex multi tenancy and internal / external architecture arrangements that might make things slightly different from expectations.

I think perhaps the best option here is that I add a config flag that allows for refreshing all known issuers when an unknown kid is seen. This would solve your problem but if used on an externally facing gateway might open you up to DoS abuse if you get a lot of invalid JWTs coming your way.

This is a quick fix and could be done by tomorrow.

[hofmannmich]

I hope your quick fix will not be needed (feels a little "dirty"). But thanks for your proposal. Maybe I will find another solution (have to talk to our networking team). I agree on your argument, that this is unusual. Fixing all this unusual behaviour brings me to another problem. The endpoint of our idp has a self signed cert as you already noticed (email). How can the plugin be configured to set a trust store for the calls to jwks and openid-config endpoint? Using insecureSkipVerify is not a good option when I set the issuers config to use a public DNS name.

[hofmannmich]

We could solve the problem from inside OpenShift to call external DNS name with https.. So fetching new kid entries should work as you described. I will have at next rotation.

The only open question is: who can I tell your plugin to use a trust store for calling these endpoints (self signed certs). I do not want to use insecureSkipVerify for production.

[agilezebra]

I'm just in the process of trying to get it to pass a RootCAs cert pool in for TLSClientConfig for the defaultClient, if supplied. So far this is not making any difference though with using a self-signed cert in the pool, even though it has CA:TRUE. I think it wants an actual root CA that was used to sign the server's cert. I'm going to try creating a custom root CA and signing with that and passing only the root CA.
Can you confirm if you have a custom root CA that you are signing these certs with or is this just a true self-signed cert?
Some thoughts though on that:

• Obviously the best solution would be that the servers don't have a self-signed cert but rather a valid cert. Is it not possible for them to create one using lets encrypt, just as traefik does?
• Note that the insecureSkipVerify is restricted to the list of hostnames that you specify, so these can be restricted to just your internal DNS names. I agree though that it should be possible to use a self-signed cert and specify this to the client to verify

[hofmannmich]

you can use the jwks-endpoint I have send you via email. https://<my-idp>/jwks
If you analyse the cert of this endpoint you can see: we have a root-ca, an intermediate ca and at last a cert for the endpoint. Some clients need the whole chain in trust store, some are fine with only the root-ca. TLS is not clear in this point. I think it is better to give the whole chain in one pem (public cert of root-ca, public cert of intermediate ca) file to your plugin.

[agilezebra]

Added https://github.com/agilezebra/jwt-middleware/releases/tag/v1.2.8 with rootCAs option. I'll email you an example with your root CA