Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds a modifier function to simplify addition of local volumes to containerop #783

Merged
merged 2 commits into from
Feb 28, 2019

Conversation

swiftdiaries
Copy link
Member

@swiftdiaries swiftdiaries commented Feb 5, 2019

The normal method of adding volumes to containerop involves 3 steps

  1. Define Volume using k8s python client
  2. Create step using containerop
  3. Add volume / volume mount to containerop

Like this:

# Define volume using the k8s python client
from kubernetes import client as k8s_client
pvc = k8s_client.V1PersistentVolumeClaimVolumeSource(claim_name='claim')
volume = k8s_client.V1Volume(name='workflow', persistent_volume_claim=pvc)
volume_mount = k8s_client.V1VolumeMount(mount_path='/mnt/workflow/',name='workflow')

# Create container_op
def step_op(mount_path, step_name='confusion_matrix'):
    return dsl.ContainerOp(
        name = step_name,
        image = 'image_name:image_tag',
        arguments = [
            'arg1', arg_1,
        ]
    )
step_name = step_op('/mnt/workflow')

# Add to containerop
step_name.add_volume(volume)
step_name.add_volume_mount(volume_mount)

This is fairly verbose.

With this change, we can add volumes from the DSL and we don't have to interact with the k8s client.
Syntax now:

  1. Create step using containerop
  2. Create volume with DSL
  3. Add volume to step using DSL modifier function
# Create container_op
def step_op(mount_path, step_name='confusion_matrix'):
    return dsl.ContainerOp(
        name = step_name,
        image = 'image_name:image_tag',
        arguments = [
            'arg1', arg_1,
        ]
    )
step_name = step_op('/mnt/workflow')

# Volume definition
import kfp.onprem as onprem
local_volume = onprem.use_local_volume('claim', 'workflow', '/mnt/workflow')

# Add to step using modifier
step_name.apply(local_volume)

This is much more readable and enables better reuse of volumes across different steps.


This change is Reviewable

@swiftdiaries
Copy link
Member Author

/cc @ramdootp

@swiftdiaries
Copy link
Member Author

/assign @Ark-kun

@vicaire
Copy link
Contributor

vicaire commented Feb 13, 2019

@swiftdiaries, what do you think of the proposal at #801?

sdk/python/kfp/onprem.py Outdated Show resolved Hide resolved
@Ark-kun
Copy link
Contributor

Ark-kun commented Feb 28, 2019

/lgtm

1 similar comment
@Ark-kun
Copy link
Contributor

Ark-kun commented Feb 28, 2019

/lgtm

@Ark-kun
Copy link
Contributor

Ark-kun commented Feb 28, 2019

/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Ark-kun

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Ark-kun

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@swiftdiaries
Copy link
Member Author

Thanks for the review @Ark-kun !

@k8s-ci-robot k8s-ci-robot merged commit 9010329 into kubeflow:master Feb 28, 2019
cheyang pushed a commit to alibaba/pipelines that referenced this pull request Mar 28, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants