Unable to use tensorboard viewer using mount path

[haibingzhao]

What steps did you take:

deploy 'ml-pipeline-ui' deployment with env:
VIEWER_TENSORBOARD_POD_TEMPLATE_SPEC_PATH=/config/view_spec_template.json

content of /config/view_spec_template.json is:

{
  "spec": {
    "containers": [{
      "volumeMounts": [{
        "name": "output",
        "mountPath": "/output"
      }]
    }],
    "volumes": [{
      "name": "output",
      "hostPath": {
        "path": "/data/output",
        "type": "Directory"
      }
    }]
  }
}

run pipeline component and finished with Unsupported storage path: /output/5/1/record/rnn error:

What happened:

switch to Artifact tab, show Unsupported storage path: /output/5/1/record/rnn error

What did you expect to happen:

no errors and show tensorboard open card as below:

Environment:

How did you deploy Kubeflow Pipelines (KFP)?
deploy kubeflow pipeline locally with k3s

KFP version: 0.5.1
KFP SDK version: 0.5.1

Anything else you would like to add:

search for source code WorkflowParser, it is confused that why not support local file schema start with '/'?

/kind bug

[Bobgy]

developer guide in https://github.com/kubeflow/pipelines/tree/master/frontend.

Would you mind trying adding a branch in source code to handle local file and if that works for you?

[Bobgy]

Thanks for the report! we haven't been supporting local volume mount for tensorboard, so this is a feature request.
/assign @jingzhang36

[haibingzhao]

developer guide in https://github.com/kubeflow/pipelines/tree/master/frontend.

Would you mind trying adding a branch in source code to handle local file and if that works for you?

I'm glad to support this feature, some small code change works now. Detail info please refer to: WorkflowParser.ts#L365 and artifacts.ts#L105

How to use?

1. clone source code, build frontend docker image, detail info: https://github.com/kubeflow/pipelines/tree/master/frontend:

cd your_pipeline_path/frontend
npm run docker

2. create config map for ml-pipeline-ui

kubectl apply -f - <<EOF
apiVersion: v1
data:
  view-spec-template.json: |-
    {
      "spec": {
          "containers": [{
            "volumeMounts": [{
              "name": "output",
              "mountPath": "/output"
            }]
          }],
          "volumes": [{
            "name": "output",
            "persistentVolumeClaim": {
              "claimName": "output"
            }
          }]
        }
    }
kind: ConfigMap
metadata:
  name: ui-view-spec-template-configmap
  namespace: kubeflow
EOF

view-spec-template.json content will be used by Viewer CRD, and tensorboad can use any path in /output directory. your tensorflow training pipeline component can attach output volume and output your data.

3. patch update ml-pipeline-ui deployment

kubectl patch deployment ml-pipeline-ui -nkubeflow --type strategic --patch '
spec:
  template:
    spec:
      containers:
      - env:
        - name: VIEWER_TENSORBOARD_POD_TEMPLATE_SPEC_PATH
          value: "/config/view_spec_template.json"
        image: gcr.io/ml-pipeline/frontend:your_build_image_version
        name: ml-pipeline-ui
        volumeMounts:
          - mountPath: /config/view_spec_template.json
            subPath: view_spec_template.json
            name: view-template-config
          - mountPath: /output
            name: output
      volumes:
        - name: view-template-config
          configMap:
            name: ui-view-spec-template-configmap
            items:
              - key: view-spec-template.json
                path: view_spec_template.json
        - name: output
           persistentVolumeClaim:
                claimName: output,
                readOnly: true
'

ml-pipeline-ui deployment attach output volume only if you want to output Confusion matrix , Markdown, ROC curve, Table, Web app data file to output volume and want to displayed at UI. if you just use TensorBoard , output volume can be removed from ml-pipeline-ui deployment.

[Bobgy]

That's awesome!
PR welcomed

[haibingzhao]

That's awesome!
PR welcomed

PR was ready: #4236

[Bobgy]

Hi @haibingzhao, we can wait for #4236 to be merged now.
Do you want to contribute to this feature's documentation too?

Because this is relatively complex, I think it probably needs its own page for documentation. I think you can start from adding a markdown documentation in this repo probably in docs/config/volume-support.md.

Later we can target moving the documentation to somewhere near https://www.kubeflow.org/docs/pipelines/sdk/output-viewer/.

[haibingzhao]

Hi @haibingzhao, we can wait for #4236 to be merged now.
Do you want to contribute to this feature's documentation too?

Because this is relatively complex, I think it probably needs its own page for documentation. I think you can start from adding a markdown documentation in this repo probably in docs/config/volume-support.md.

Later we can target moving the documentation to somewhere near https://www.kubeflow.org/docs/pipelines/sdk/output-viewer/.

ok, I will document this later.

[haibingzhao]

@Bobgy I submit a PR to document this at: #4345, any suggestion will be welcome

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.