Kubeflow Pipelines Volume examples

[sarahmaddox]

The Kubeflow Pipelines DSL was recently extended to include PipelineVolume and PipelineVolumeSnapshot. These are an abstraction for the user to manipulate PVCs and VolumeSnapshots for data management purposes, also exposing dependencies according to their usage.

See PR kubeflow/pipelines#926 for further details.

We need and example and doc covering the above feature.

Thanks @dansanche for suggesting this doc!

[issue-label-bot]

Issue-Label Bot is automatically applying the label improvement/enhancement to this issue, with a confidence of 0.82. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[elikatsis]

Hi there!
The initial design has changed after the review process.
For example, PipelineVolumeSnapshots don't exist. Also, the manipulation of any Kubernetes Resource is available via ResourceOp class.

The final implementation is closer to this design. The merged PR has just a few minor enhancements (it doesn't diverge from the logic described).

There are some examples under https://github.com/kubeflow/pipelines/tree/master/samples/resourceops.
However, I will be refactoring the directories of the samples to group the ones that aim to show PVC manipulation and those which refer to the VolumeSnapshots to make the navigation easier for the users.

I'll be happy to help with the documentation!

[sarahmaddox]

Thanks @elikatsis! Would you like to take part in the doc sprint, to help create the docs? July 10-12. You can take part on site (Kirkland) or online.

Here's the Kanban board for the Kubeflow Doc Sprint.

Full details are on the doc sprint wiki.

Here's the signup form.

[elikatsis]

@sarahmaddox, Hello Sarah !

Unfortunately, I couldn't make it for this doc sprint and I am really sorry I didn't mention it earlier.
I've already begun working on some basic documentation with the PR kubeflow/pipelines#1433 (I'll ping the PR thread to see what's need to be done to get it merged).

I believe I will be able to contribute to the documentation within the next few weeks!

[sarahmaddox]

/assign @elikatsis

Thanks @elikatsis it's great to know that you'll be able to tackle the docs. No worries that it won't be within the timeframe of the doc sprint.

[elikatsis]

Hello!

Right now we've got #837 merged (link to the official website) and there are also several examples in kubeflow/pipelines repo:

• samples/core/resource_ops
• samples/core/volume_ops
• samples/core/volume_snapshot_ops
• samples/contrib/volume_ops
• samples/contrib/volume_snapshot_ops

There are some READMEs, too (1, 2).

I think all of the above close this issue.

@sarahmaddox, Sarah, what's your opinion? Is there anything else you think we should do?

[sarahmaddox]

Thanks @elikatsis! The information is a little scattered at the moment. We have:

• The SDK overview on the website which links to the VolumeOp etc in the SDK reference docs.
• The SDK reference docs which have very limited info.
• The READMEs and samples that you've linked above.

It'd be good to have an overview of the features on the website, in a page under this section: https://www.kubeflow.org/docs/pipelines/sdk/
The overview should explain what the features allow users to do, and link to the samples for indepth usage info.

Preferably the info in the READMEs should move to the website, as they contain user-focused info rather than developer-focused info.