KFP SDK should support gcloud credential

[hongye-sun]

This is required when user wants to talk to a remote KFP endpoint which is protected by IAP from a local environment.

The current support we have is to look for GCE service account credentials in the machine and send them to pass the IAP auth. We also need to support to use user's gcloud credential to support local command line experience.

[Ark-kun]

/cc @kevinbache

[jlewi]

For GCP IAP see here for how to obtain a JWT and attach it to the request.
https://cloud.google.com/iap/docs/authentication-howto

You can't use the gcloud credential e.g. gcloud auth print-access-token.

This is by design because gcloud has a credential with cloud platform scope whereas with IAP a user is accessing web services separate from Google cloud.

[IronPan]

Update -
The python client library need to expose the option to set the request header.
https://github.com/kubeflow/pipelines/blob/master/sdk/python/kfp/_client.py#L226

[Ark-kun]

The python client library need to expose the option to set the request header.

Can you elaborate? Dose the client need to add Authorization: Bearer to all requests?

[Ark-kun]

Update -
The python client library need to expose the option to set the request header.
https://github.com/kubeflow/pipelines/blob/master/sdk/python/kfp/_client.py#L226

We're already setting the header here: https://github.com/kubeflow/pipelines/blob/master/sdk/python/kfp/_client.py#L64

The Client class has client_id paramater that's used to fetch the token: https://github.com/kubeflow/pipelines/blob/master/sdk/python/kfp/_auth.py#L28

Do we need to add something extra?

[hongye-sun]

The current implementation only supports auth from service account. In order to support auth from desktop, we will need to implement a flow which is similar with cloud sdk:

1. Allow user to provide IAP client id and other client id and secret.
2. Print out a link which will open a browser for user to login with google account and the code can get the authorization code. (Not sure how cloud sdk supports to get data from browser, but it should be doable).
3. Use other client id, secret, auth code to get the refresh token
4. Persist refresh token to local disk
5. Use IAP client id, other client id, other client secret and refresh token to exchange an ID token, which may only last for a short period.
6. Set the ID token in header to each request to KFP endpoint until it expires, then repeat from step 5.

It seems not trivial to implement. Not sure if there is an easier way to support the whole flow.

[Ark-kun]

2. Print out a link which will open a browser for user to login with google account and the code can get the authorization code. (Not sure how cloud sdk supports to get data from browser, but it should be doable).

Isn't this what gcloud auth login is doing?
It then persists the credentials locally and allows some tools to use them.

[daikeshi]

Is there any plan to add gcloud credential support for IAP endpoint? We are building a managed kubeflow cluster for multiple teams to use. It'll be super helpful if glcoud credential is supported for the IAP endpoint. OW, teams will have to download a service account key to connect kfp.Client, or we will need to grant them kubernetes engine developer role. If there's anything I can help please let me know. I'm happy to contribute! Thanks!

[hongye-sun]

/reopen

Reopen the bug to support IAP auth token from desktop. The original fix can only support if user is in k8s developer role, which doesn't work for "managed kubeflow" use case.

@daikeshi, the details on how to get auth token is in https://cloud.google.com/iap/docs/authentication-howto#authenticating_from_a_desktop_app.

Please refer to me previous comment on a rough idea on how to get it support in SDK. It seems non-trivial.

[k8s-ci-robot]

@hongye-sun: Reopened this issue.

In response to this:

/reopen

Reopen the bug to support IAP auth token from desktop. The original fix can only support if user is in k8s developer role, which doesn't work for "managed kubeflow" use case.

@daikeshi, the details on how to get auth token is in https://cloud.google.com/iap/docs/authentication-howto#authenticating_from_a_desktop_app.

Please refer to me previous comment on a rough idea on how to get it support in SDK. It seems non-trivial.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[daikeshi]

@hongye-sun thanks! It does seem complicated. I'll give a try and see how much I can help with :-)

[jlewi]

What does gcloud credential mean?

Do you mean using gcloud auth print access-token? The access token generated by gcloud won't work with IAP enabled endpoints; this is by design.

You can use IAP with end user credentials and don't need to use a service account. I think the way this works is the SDK should generate a JWT to attach to requests. To generate the JWT you direct the user through the OAuth web flow.

[jlewi]

@IronPan Is this issue still relevant? I believe you tested 0.6 with IAP and were able to successfully submit pipelines (see #1080) via the SDK using IAP.

[gaoning777]

It is supported in #2626.
Feel free to reopen the issue.