Frontend - Add support for artifacts stored in S3

[Jeffwan]

I was working on more storage options for artifacts. Original idea was to add local and s3 for no-prem and aws users. Notice local is a little bit tricky because pipeline-UI can not access PVC actively. After some discussion, we think ViewerCRD or minio would be choice for on-prem users.

S3 parts, I notice @DmitryBe has some work #767. I can not reach out to him and wait for few days. I borrow some code snippet and reuse minio client in this PR.

I will have some corresponding changes in kubeflow/kubeflow later to mount AWS credential to ui pod.

---

This change is 

[k8s-ci-robot]

Hi @Jeffwan. Thanks for your PR.

I'm waiting for a kubeflow member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

Hi @Jeffwan. Thanks for your PR.

I'm waiting for a kubeflow member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[rileyjbauer]

/ok-to-test

[rileyjbauer]

LGTM! Thank you for this PR!

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rileyjbauer

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• frontend/OWNERS [rileyjbauer]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[cdyangzhenyu]

hi @Jeffwan, Can support the other s3 API compatible storage such as ceph/minio?

[Jeffwan]

hi @Jeffwan, Can support the other s3 API compatible storage such as ceph/minio?

I think minio support has been added. It will work for S3 API compatible storage. The only different is s3 endpoints.

[khani92]

How can I get the AWS creds set as env var in the ml-pipeline-ui pod? @Jeffwan , any thoughts?

[Jeffwan]

How can I get the AWS creds set as env var in the ml-pipeline-ui pod? @Jeffwan , any thoughts?

You can attach aws env in pipeline-ui project. something like this

containers:
    image: gcr.io/ml-pipeline/frontend:0.1.23
    env:
       - name: AWS_ACCESS_KEY_ID
         valueFrom:
           secretKeyRef:
             key: AWS_ACCESS_KEY_ID
             name: aws-secret
       - name: AWS_SECRET_ACCESS_KEY
         valueFrom:
           secretKeyRef:
             key: AWS_SECRET_ACCESS_KEY
             name: aws-secret

[Shasvat0601]

@Jeffwan
Could you please explain where to make the above snippet to attach aws env in pipeline-ui project?
do we write this in podtemplatespec?

[Jeffwan]

@Shasvat0601

You just need to attach AWS credential ENV in pipeline ui deployment. Either use plain text or read from secret.

containers:
    image: gcr.io/ml-pipeline/frontend:0.1.23
    env:
       - name: AWS_ACCESS_KEY_ID
         valueFrom:
           secretKeyRef:
             key: AWS_ACCESS_KEY_ID
             name: aws-secret
       - name: AWS_SECRET_ACCESS_KEY
         valueFrom:
           secretKeyRef:
             key: AWS_SECRET_ACCESS_KEY
             name: aws-secret

[Shasvat0601]

@Jeffwan Thank you for your reply. really appreciate it!
I am attaching the AWS credentials ENV in pipeline UI by mounting the secret as a volume
Still I get 500 error: access denied

What can be causing this?

Containers:

ml-pipeline-ui:

Container ID: docker://415762af72b61d51503cf923f10904ad9ec9d5dd1e84d2b13d9ca812f664dfee
Image:          gcr.io/ml-pipeline/frontend:0.1.31
Image ID:       docker-pullable://gcr.io/ml-pipeline/frontend@sha256:7022552d3481edab0e683f9fcdad2ee29f2cabcfd54536d63dfaf20c101a6ed6
Port:           3000/TCP
Host Port:      0/TCP
State:          Running
  Started:      Mon, 09 Dec 2019 18:01:29 -0500
Ready:          True
Restart Count:  0
Environment:    <none>
Mounts:
  /var/run/secrets/kubernetes.io/serviceaccount from ml-pipeline-ui-token-265pv (ro)
Volumes:
  ml-pipeline-ui-token-265pv:
  Type:        Secret (a volume populated by a Secret)
  SecretName:  ml-pipeline-ui-token-265pv
  Optional:    false

[Jeffwan]

500 error: access denied

You mean you get 50x when you visit web page? could you share logs of front UI?

[Shasvat0601]

@Jeffwan
UI-pod has mounted secret as a volume

1. UI pod first accesses minio for metadata - which is sucessful
2. then UI-pod tries to access s3- because source of my viz is in s3

    metadata = {
      'outputs': [{
        'type': 'confusion_matrix',
        'format': 'csv',
        'schema': [
          {'name': 'target', 'type': 'CATEGORY'},
          {'name': 'predicted', 'type': 'CATEGORY'},
          {'name': 'count', 'type': 'NUMBER'},
        ],
        'source': s3-bucket/path/confusion_matrix.csv

[Shasvat0601]

@Jeffwan
Failed to get object in bucket s3-bucket at path s3-path: S3Error: Access Denied

[Jeffwan]

@Shasvat0601 em.. sorry for late. Could you please open a new issue and we can track and resolve there?

[Shasvat0601]

@Jeffwan sure!
Thank you :)

[dutingda]

@Jeffwan If my s3 artifacts are within html, is that also supported?