Using admin-gcp-sa and user-gcp-sa for different ContainerOp in same pipeline doesn't work

[aakashbajaj]

Using Kubeflow pipeline SDK, I'm preparing following components where first container requires user service account and second container requires admin service account (as it creates deployment and services using kubectl in the underlying cluster).

tensorboard = dsl.ContainerOp(
    name='tensorboard',
    image=SOME_IMAGE_PATH,
    arguments=[
....
      ],
      ).apply(gcp.use_gcp_secret('user-gcp-sa'))  

  tfserve = dsl.ContainerOp(
    name='tfserve',
    image=SOME_IMAGE_PATH,
    arguments=[
....
      ],
      ).apply(gcp.use_gcp_secret('admin-gcp-sa'))

But the generated source yaml file for the pipeline contains only one user-gcp-sa volume mount for the workflow. I have checked and there is no other admin-gcp-sa mount in the file.

  volumes:
  - name: gcp-credentials
    secret:
      secretName: user-gcp-sa

It causes the tfserve module to fail as user-gcp-sa doesn't enough permissions to create deployments in the cluster.

[aakashbajaj]

Following is secret volume mount in same pipeline yaml file.
tfserve was mentioned to use admin-gcp-sa but is provided with user-gcp-sa

env:
      - name: GOOGLE_APPLICATION_CREDENTIALS
        value: /secret/gcp-credentials/user-gcp-sa.json
      - name: CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE
        value: /secret/gcp-credentials/user-gcp-sa.json
      image: gcr.io/speedy-aurora-193605/retina-tfserve:latest
      volumeMounts:
      - mountPath: /secret/gcp-credentials
        name: gcp-credentials

[Ark-kun]

This is a bug. Currently the DSL compiler does not properly de-duplicate volumes.

pipelines/sdk/python/kfp/compiler/compiler.py

Line 504 in 2a9bbdf

#TODO: check for duplicity based on the serialized volumes instead of just name.

I'll add a workaround which will use the secret name as volume name.

If you want to unblock yourself right now, please copy and modify the code of use_gcp_secret to generate different volume names.

[gaoning777]

Also, check your IAM and you will find an admin service account created during the deployment. The service account is called DEPLOYMENT_NAME-user@PROJECT_NAME.iam.gserviceaccount.com