Concierge credential exchange fails because of an outdated ID token

[rdimitrov]

What happened?

I have a local Kind cluster setup running Concierge v0.10.0 configured with Dex v2.30.0. When I have a clean install it works okay, but once I recreate that same setup I get an authentication failure. By recreating I mean deleting the old cluster, creating a new one with Concierge and Dex redeployed, and also regenerating the CA bundle used by Dex.

I’m able to get the config file using the Pinniped CLI without issues, but it seems that there's a problem with verifying the ID Token I get from Dex.

The errors I’m seeing are -

kubectl --kubeconfig my-cluster.yaml get pods -n auth-with-dex
Error: could not complete Concierge credential exchange: login failed: authentication failed
Unable to connect to the server: getting credentials: exec: executable /usr/local/bin/pinniped failed with exit code 1

and the logs from the Concierge pod are showing -

I0810 16:51:07.955524       1 trace.go:205] Trace[1124895541]: "create" kind: (10-Aug-2021 16:51:07.946) (total time: 8ms):
Trace[1124895541]: ---"failure" failureType:token authentication,msg:oidc: verify token: failed to verify signature: failed to verify id token signature 8ms (16:51:00.955)
Trace[1124895541]: [8.718934ms] [8.718934ms] END

What did you expect to happen?

I should have been redirected to Dex where I can log in.

What is the simplest way to reproduce this behavior?

Recreate an environment without changing any parameters and clearing the pinniped cache folder.

In what environment did you see this bug?

• Pinniped server version: v0.10.0
• Pinniped client version: v0.10.0
• Pinniped container image (if using a public container image): projects.registry.vmware.com/pinniped/pinniped-server:v0.10.0@sha256:3bdfb9ad9275449f07614081eca27cff16f15562aeabfee9214a0b0506bb6320
• Pinniped configuration (what IDP(s) are you using? what downstream credential minting mechanisms are you using?): Dex v2.30.0
• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.0", GitCommit:"c2b5237ccd9c0f1d600d3072634ca66cefdf272f", GitTreeState:"clean", BuildDate:"2021-08-04T17:56:19Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-06-22T04:47:11Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"darwin/amd64"}

• Kubernetes installer & version (e.g., kubeadm version): kind image version - kindest/node:v1.21.1
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a): darwin 20.6.0
• Others:

What else is there to know about this bug?

Clearing the ~/.config/pinniped folder fixes the issue.

[enj]

Clearing the ~/.config/pinniped folder fixes the issue.

I think the pinniped CLI could do better in terms of detecting when its cached data is not valid (i.e. prompt the user to login again when the cached creds fail during the token exchange).

[cfryanr]

It feels like the CA bundle of the OIDC issuer should be part of the cache key in sessions.yaml. If the client is talking to a server with a different CA bundle, then it is effectively talking to a different server.