[GCP] pipelines supports workload identity

[jlewi]

We would like to support using workload identity on GCP.

See kubeflow/kubeflow#3638

Opening this bug to track adding support for workload identity with KFP

A couple questions

• Which namespaces will KFP pipelines running in? Are users still running in the "kubeflow" namespace or can pipelines run in other namespaces e.g. "kf-${USER}"

• Does KFP support specifying which K8s service account is used by Argo pods?

  • The K8s service account will determine which GCP service account is used.
  • Current pattern in Kubeflow is that the profile resource will create a namespace with a default-editor K8s service account which would be bound to an appropriate GCP service account
  • So pipelines might be able to use that KSA (K8s Service account by default.)

/cc @jessiezcc
/cc @IronPan

[jessiezcc]

Pipeline namespace is configurable, #304 tracked the ask and this PR addressed it.
As for service account, pipeline runs under user-gcp-sa which reflects the default active service account based on #374. @IronPan can confirm.

[jlewi]

@IronPan @jessiezcc Does Kubeflow pipelines plan to support workload identity?

Here are the docs for workload identity.

With workload identity Kubernetes Service Accounts (KSA) can be bound to GCP Service Accounts (GSA).

So with workload identity pipeline users would probably want the ability to specify what service account the pods in the pipeline use as that would determine what GSA the pod uses.

[gaoning777]

Yes. We will support the workload identity within the multi-user efforts in the Q4 such that users are authenticated to use GCP resources.

[Bobgy]

Note, one work item:

• pipeline-ui needs gcs read permission when using workload identity mlpipeline-ui-metadata html output not displaying on v0.7rc6 (worked on 0.6.2) #2501, quick fix is done in Fix ml-pipeline-ui doesn't have gcp permission bug manifests#594 without workload identity.

[IronPan]

+1 we are planning on migrating the full stack to WI. Thanks Jeremy.

[IronPan]

We also need clean documentations how to use WI when authoring the pipeline component, after WI is completed.

[Bobgy]

Work items for standalone deployment:

• Migrate standalone deployment to workload identity on GCP Migrate standalone deployment to workload identity on GCP #2619
• Deploy doc in https://www.kubeflow.org/docs/pipelines/installation/standalone-deployment/ and Pipelines GCP authentication doc in https://www.kubeflow.org/docs/gke/authentication-pipelines/, PR: Introducing workload identity setup for pipelines standalone website#1489
• Document which service accounts we use and how we can configure their IAM permissions.
• Test flakiness due to workload identity issue, PR: Use new K8s version to improve workload identity stability #2777
• Standalone deployment with managed storage should support workload identity too: [Standalone] env/gcp manifest should support workload identity #3377

[Bobgy]

Work items for kubeflow deployment:

• GCP authentication doc in https://www.kubeflow.org/docs/gke/authentication/#authentication-from-kubeflow-pipelines, also addressed in Introducing workload identity setup for pipelines standalone website#1489
• pipeline-runner, ml-pipeline-ui and other KSAs that are expected to use GCP APIs should be bound to GSA by default.
• Workaround: use kf-user KSA for pipelines and tensorboard. KFP builtin samples fail in KF1.0 kubeflow#4803 and TensorBoard KFP viz doesn't appear to work properly with KF 1 installation kubeflow#4795
• ml-pipeline-ui deployment should no longer mount user-gcp-sa secret: source
• Document how to use workload identity with Full Kubeflow deployment: [KFP] Update authenticating to GCP doc about KF 1.0 and hosted website#1832

[Bobgy]

Work items not specific to any deployments:

• [Sample tests] Upgrade GCP sdk to 272 in sample test
  

• other component/sample docs scattered in the repo should refer to https://www.kubeflow.org/docs/gke/authentication-pipelines/, PR: [Doc] Change sample/component/sdk documentation to not use use_gcp_secret #2782

• SDK doc in https://kubeflow-pipelines.readthedocs.io/en/latest/source/kfp.extensions.html#kfp.gcp.use_gcp_secret

• SDK/backend should expose interface to allow using a different KSA [SDK/Backend/UI] Configurable service account #3415

• parameterized tfx oss sample should stop using use_gcp_secret by default: https://github.com/kubeflow/pipelines/blob/master/samples/core/parameterized_tfx_oss/parameterized_tfx_oss.py#L153

• container builder should use its own KSA kfp _generate_kaniko_spec permissions under workload identity #2814

• better UI error message for missing user-gcp-sa [UI] Better error message on missing user-gcp-sa #3112

• tensorboard should no longer use default service-account TensorBoard KFP viz doesn't appear to work properly with KF 1 installation kubeflow#4795

[Bobgy]

Work items blocking multi user support and hosted are done. Changed to p1 priority.

[jlewi]

@Bobgy is it possible for a user to set the KSA when using TFX?

/cc @ucdmkt

[numerology]

@Bobgy is it possible for a user to set the KSA when using TFX?

/cc @ucdmkt

@jlewi
If you're talking about using KubeflowDagRunner, currently it's not supported, but I think it's definitely doable.

Is there any use case/feature depending on this? Thanks!

[Bobgy]

Work Items for GCP hosted deployment:

• Documentation of how to set up Workload Identity: [KFP] Update authenticating to GCP doc about KF 1.0 and hosted website#1832

[jlewi]

@numerology I think for TFX to work with workload identity TFX (KubeflowDagRunner) will need to support setting the Kubernetes Service Account (KSA) that steps in the pipeline use. The KSA will control which Google Service Account is used by that step. So I think it will be important for that to be supported. Do you want to file a separate issue for that?

[numerology]

@jlewi
Sounds good. However it depends on the ability of specifying the KSA in use through KFP SDK, which is currently missing. @Bobgy can correct me if I'm wrong, I think it's the planned as a "phase 3" work.

[Bobgy]

@numerology Yes, that's right.
But in phase 2, we'd support choosing KSA in KFP API at pipeline level. @jlewi will that be enough for your use case?

[jlewi]

@Bobgy I think setting a per pipeline KSA is fine.

[Ark-kun]

However it depends on the ability of specifying the KSA in use through KFP SDK, which is currently missing.

Can we set that on cluster-level instead (e.g. using a configmap)? I'd prefer not to have any account-specific information in the pipeline file.

[Bobgy]

Choosing a KSA when running a pipeline is suppported now.