401 Unauthorized with oidc user.

[pnathan]

Hi,

I'm getting a consistent 403 forbidden. This doesn't happen with the Python library or kubectl. The user is an oidc user, backed by Auth0. I have ensured that the token is valid.

I can't see any particular place where I need to pass in or set a flag. The cluster is a kops-generated cluster with its own generated certificate.

Looking at https://github.com/kubernetes-client/javascript/blob/master/src/oidc_auth.ts#L43 , I see that I have a flag in my config file extra-scopes: openid profile, which doesn't appear to be an option in the dict there. But this doesn't seem to matter in the Python library - https://github.com/kubernetes-client/python-base/blob/6b6546131217a2a9fdcf431a286c346619d2923a/config/kube_config.py#L289

First, the package.json:

    "dependencies": {
        "@kubernetes/client-node": "^0.11.0",
[snip]
        "kubernetes-client": "^0.11.0",

And for an example function:

function Example() {
    console.log(kubeconfig.currentContext); // this is correctly returned
    console.log("User", kubeconfig.getCurrentUser());   // this is the correct user object.
    console.log("Context", kubeconfig.getContextObject('Redacted'));  // This corresponds to the ~/.kube/config file.

    k8sApi.listNamespacedPod('default')
        .then((res:any) => {
            console.log(res.body.response);
        })
        .catch((error: any) => {
            // Yet, here we have 401.
            console.log("Error: ", error.response.body);
        });
}

I'm somewhat stumped here, I'm afraid.

[brendandburns]

Can you provide an (anonymized) example of what your kubeconfig file looks like?

Thanks!

[pnathan]

Absolutely. Here's a kubeconfig dump, along with a redacted program.

minified

$ kubectl config view --minify
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://api.example.com
  name: human-name
contexts:
- context:
    cluster: human-name
    namespace: default
    user: a-user
  name: human-name
current-context: human-name
kind: Config
preferences: {}
users:
- name: a-user
  user:
    auth-provider:
      config:
        client-id: a-client-id
        client-secret: it-is-very-secret
        extra-scopes: openid profile
        id-token: jwt-token-here
        idp-issuer-url: https://auth0-redirect.example.com/
        refresh-token: ""
      name: oidc

program

#!/usr/bin/env node

const k8s = require('@kubernetes/client-node');

function k8sfunc () {
    const kubeconfig = new k8s.KubeConfig();
    kubeconfig.loadFromDefault();
    const k8s_url = kubeconfig.getCurrentCluster().server;
    const k8sApi = kubeconfig.makeApiClient(k8s.CoreV1Api);

    k8sApi.listNamespacedPod('default')
        .then((res:any) => {
            console.log(res.body.response);
        })
        .catch((error: any) => {
            // Yet, here we have 401.
            console.log("Error: ", error.response.body);
        });
}

k8sfunc();

packaging.

package.json

{
    "name": "thing",
    "version": "1.0.0",
    "description": "thing",
    "dependencies": {
        "@kubernetes/client-node": "^0.11.0"
    },
    "devDependencies": {
        "@types/node": "^12.7.12",
        "nodemon": "^1.19.3",
        "ts-node": "^8.4.1",
        "typescript": "^3.6.4"
    },
    "scripts": {
        "start": "nodemon --watch 'src/**/*.ts' --exec 'ts-node' src/index.ts",
        "build": "tsc -p ."
    },
    "repository": {},
    "main": "./lib/index.js",
    "bin": {
        "thing": "./lib/index.js"
    },
    "private": true
}

typescript config

tsconfig.json

{
    "compilerOptions": {
        "target": "es6",
        "module": "commonjs",
        "lib": ["es6", "es2015", "dom", "es2019"],
        "declaration": true,
        "outDir": "lib",
        "rootDir": "src",
        "strict": true,
        "types": ["node"],
        "esModuleInterop": true,
        "resolveJsonModule": true
    },
}

other bits:

Linux pnathan1 4.9.0-11-amd64 #1 SMP Debian 4.9.189-3+deb9u2 (2019-11-11) x86_64 GNU/Linux

nvm use v10.17.0

end output vs kubectl

$ npm run-script start

> thing@1.0.0 start /home/pnathan/external-src/k8s-login-uh-oh
> nodemon --watch 'src/**/*.ts' --exec 'ts-node' src/index.ts

[nodemon] 1.19.4
[nodemon] to restart at any time, enter `rs`
[nodemon] watching dir(s): src/**/*.ts
[nodemon] watching extensions: ts,json
[nodemon] starting `ts-node src/index.ts`
Error:  {
  kind: 'Status',
  apiVersion: 'v1',
  metadata: {},
  status: 'Failure',
  message: 'Unauthorized',
  reason: 'Unauthorized',
  code: 401
}
[nodemon] clean exit - waiting for changes before restart
^C

$ kubectl get po -n default
<some pods>

[brendandburns]

Is your refresh token truly empty string ("") as in the example config?

If so, I think the problem is this line here:
https://github.com/kubernetes-client/javascript/blob/master/src/oidc_auth.ts#L46

Which will bail before we even get to evaluating the token. This seems like a bug to me regardless, so I'll send a PR to fix, but a quick test would be for you to change the refresh token to a not-empty string and see if that fixes things.

If you really have a refresh value, then I'll need to keep investigating.

[pnathan]

(1) Good news: I do not. It really is empty!

(2) The results of the test are...

Changing the refresh token to "yolo"

        refresh-token: yolo

I validate I can kubectl get po - pods are gotten.

then:

(node:30212) [DEP0066] DeprecationWarning: OutgoingMessage.prototype._headers is deprecated
Error:  {
  error: 'unauthorized_client',
  error_description: "Grant type 'refresh_token' not allowed for the client.",
  error_uri: 'https://auth0.com/docs/clients/client-grant-types'
}

which is fascinating. Since I have had it for a while and kubectl uses it happily.

I then remove it altogether from this specific user, and the following results:

Error:  {
  kind: 'Status',
  apiVersion: 'v1',
  metadata: {},
  status: 'Failure',
  message: 'Unauthorized',
  reason: 'Unauthorized',
  code: 401
}

I validate that refresh-token's presence entirely is not required for the kubectl to succeed - kubectl works along happily enough.

[brendandburns]

So what is happening (I think) is that the javascript client tries to refresh first when there's no expiration timestamp. I think the kubectl (and Python) clients might simply try to use the token that's in the file.

I think that probably trying the token makes sense so I'll update the Javascript client to use them.

Also, I think the issue might be that refresh for that kubeconfig is broken. I need to dig into the OIDC client code/protocol a little more to truly understand what is the correct expectations here with regards to refresh token handling.

[pnathan]

Let me know if you want me to test a patch. I'm also somewhat surprised that an empty token is accepted (perhaps kubectl doesn't send empty tokens?).

[brendandburns]

Some observations from the kubectl oidc code

Extra scopes is deprecated and unused:
https://github.com/kubernetes/client-go/blob/16bcffe0e4b7c58035b4d0debb40717016567ce4/plugin/pkg/client/auth/oidc/oidc.go#L128

It does require a non-zero length refresh token:
https://github.com/kubernetes/client-go/blob/16bcffe0e4b7c58035b4d0debb40717016567ce4/plugin/pkg/client/auth/oidc/oidc.go#L239

But it first pulls the expired date from the the id token:
https://github.com/kubernetes/client-go/blob/16bcffe0e4b7c58035b4d0debb40717016567ce4/plugin/pkg/client/auth/oidc/oidc.go#L333

The Javascript code doesn't do this last part, which is why (I think) it's not working for your client.

I think I have enough information now to fix this particular issue. (and I've learned some new things about OIDC too :)

[brendandburns]

I have a PR that I believe will fix this here:

#388

If you can patch and validate it would be much appreciated.

[pnathan]

Hi,

Sorry about the delay.

Failure in a new and improved way.

Error:  SyntaxError: Invalid padding
    at parse (/home/pnathan/external-src/k8s-login-uh-oh/node_modules/rfc4648/lib/index.cjs.js:22:11)
    at Object.parse$1 [as parse] (/home/pnathan/external-src/k8s-login-uh-oh/node_modules/rfc4648/lib/index.cjs.js:169:12)
    at Function.expirationFromToken (/home/pnathan/external-src/k8s-login-uh-oh/node_modules/@kubernetes/client-node/dist/oidc_auth.js:17:45)
    at OpenIDConnectAuth.<anonymous> (/home/pnathan/external-src/k8s-login-uh-oh/node_modules/@kubernetes/client-node/dist/oidc_auth.js:58:65)
    at Generator.next (<anonymous>)
    at /home/pnathan/external-src/k8s-login-uh-oh/node_modules/tslib/tslib.js:110:75
    at new Promise (<anonymous>)
    at Object.__awaiter (/home/pnathan/external-src/k8s-login-uh-oh/node_modules/tslib/tslib.js:106:16)
    at OpenIDConnectAuth.refresh (/home/pnathan/external-src/k8s-login-uh-oh/node_modules/@kubernetes/client-node/dist/oidc_auth.js:56:24)
    at OpenIDConnectAuth.<anonymous> (/home/pnathan/external-src/k8s-login-uh-oh/node_modules/@kubernetes/client-node/dist/oidc_auth.js:52:25)

when refresh-token is either "ABCD", "", or, simply not present.

https://github.com/swansontec/rfc4648.js/blob/master/src/codec.ts#L33 is the error.

FYI: the ID token is JWT, and there's an example parser at https://jwt.io/

[brendandburns]

Does your JWT parse correctly in the parser at https://jwt.io?

[brendandburns]

Can you change

const payload = base64url.parse(parts[1]);

to:

const payload = base64url.parse(parts[1], { loose: true } as any);

In the patch and see if it works?

[pnathan]

Does your JWT parse correctly in the parser at https://jwt.io?

Yes.

Hotwiring the patch in:

   `const payload = rfc4648_1.base64url.parse(parts[1], {loose: true });`

that works! as any fails syntactically with my versions of things.

[brendandburns]

ok, thanks for the confirmation. I will update my PR with that change.