Migrate volume and loop unit tests to component.yaml and verify with …

…v2 (kubeflow#692) * migrate unit tests to component.yaml and verify with v2 * update readme to remove volumesnapshot * address comments
red-hat-data-services · Aug 10, 2021 · 8d8d2b3 · 8d8d2b3
1 parent 9d4643b
commit 8d8d2b3
Show file tree

Hide file tree

Showing 38 changed files with 1,022 additions and 1,540 deletions.
diff --git a/sdk/FEATURES.md b/sdk/FEATURES.md
@@ -26,7 +26,7 @@ and test pipelines found in the KFP repository.
   - [Features with the Same Behavior as Argo](#features-with-the-same-behavior-as-argo)
     - [InitContainers](#initcontainers)
     - [Conditions](#conditions)
-    - [ResourceOp, VolumeOp, and VolumeSnapshotOp](#resourceop-volumeop-and-volumesnapshotop)
+    - [ResourceOp and VolumeOp](#resourceop-and-volumeop)
     - [Output Parameters](#output-parameters)
     - [Input Artifacts](#input-artifacts)
     - [Output Artifacts](#output-artifacts)
@@ -214,17 +214,10 @@ is an example of how to use this feature.
 Conditions are used for determining whether to execute certain components based on the output of the condition checks. In KFP Argo, each condition is represented as an Argo DAG template so it can be used as a dependency for other Argo templates. To replicate this in KFP Tekton, we put our condition into a dedicated Tekton task so that conditions can be treated as a dependency for other Tekton tasks. Another advantage of creating conditions using Tekton tasks is that we can have more flexible conditions such as comparing an integer and a float number, which currently is not available in Tekton. We are using the Tekton [when expression](https://github.com/tektoncd/pipeline/blob/master/docs/pipelines.md#guard-task-execution-using-whenexpressions) to check whether the condition task has succeeded or not. We created a custom python image to replicate the same condition checks that are in Argo and made it as the default in our compiler. The
 [flip-coin](/samples/flip-coin) example demonstrates how to use multiple conditions within the same pipeline.
 
-### ResourceOp, VolumeOp, and VolumeSnapshotOp
+### ResourceOp and VolumeOp
 
-[ResourceOp, VolumeOp, and VolumeSnapshotOp](https://www.kubeflow.org/docs/pipelines/sdk/manipulate-resources/) are special operations for
-creating Kubernetes resources on the pipeline cluster. ResourceOp is a basic operation for manipulating any Kubernetes resource. VolumeOp
-and VolumeSnapshotOp are operations for creating a unique Volume/VolumeSnapshot per pipeline and can be used as volume/snapshot with any pipeline ContainerOp. Because Tekton has no support for these features natively, we have to implement our own custom task called kubectl-wrapper for creating these Kubernetes resources. The [resourceop_basic](/sdk/python/tests/compiler/testdata/resourceop_basic.py),
-[volume_op](/sdk/python/tests/compiler/testdata/volume_op.py), and
-[volume_snapshot_op](/sdk/python/tests/compiler/testdata/volume_snapshot_op.py) python tests are examples of how to use these features.
-
-One thing to pay attention to is that VolumeSnapshot won't be available by default on all the supported Kubernetes versions. Therefore,
-[VolumeSnapshotDataSource feature gate](https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/) must be enabled
-in order to use VolumeSnapshotOp.
+[ResourceOp and VolumeOp](https://www.kubeflow.org/docs/pipelines/sdk/manipulate-resources/) are special operations for
+creating Kubernetes resources on the pipeline cluster. ResourceOp is a basic operation for manipulating any Kubernetes resource. VolumeOps are operations for creating a unique Volume per pipeline and can be used as volume with any pipeline ContainerOp. Because Tekton has no support for these features natively, we have to implement our own custom task called kubectl-wrapper for creating these Kubernetes resources. The [resourceop_basic](/sdk/python/tests/compiler/testdata/resourceop_basic.py) and [volume_op](/sdk/python/tests/compiler/testdata/volume_op.py) python tests are examples of how to use these features.
 
 ### Output Parameters
 

diff --git a/sdk/python/tests/compiler/compiler_tests.py b/sdk/python/tests/compiler/compiler_tests.py
@@ -388,13 +388,6 @@ def test_volumeOp_workflow(self):
     from .testdata.volume_op import volumeop_basic
     self._test_pipeline_workflow(volumeop_basic, 'volume_op.yaml')
 
-  def test_volumeSnapshotOp_workflow(self):
-    """
-    Test compiling a volumeSnapshotOp basic workflow.
-    """
-    from .testdata.volume_snapshot_op import volume_snapshotop_sequential
-    self._test_pipeline_workflow(volume_snapshotop_sequential, 'volume_snapshot_op.yaml')
-
   def test_hidden_output_file_workflow(self):
     """
     Test compiling a workflow with non configurable output file.

diff --git a/sdk/python/tests/compiler/testdata/volume.py b/sdk/python/tests/compiler/testdata/volume.py
@@ -15,18 +15,53 @@
 
 import kfp.dsl as dsl
 from kubernetes.client import V1Volume, V1SecretVolumeSource, V1VolumeMount, V1EnvVar
+from kfp import components
+
+OP1_STR = """
+name: download
+outputs:
+  - {name: downloaded, type: String}
+implementation:
+  container:
+    image: google/cloud-sdk
+    command:
+    - sh
+    - -c
+    args:
+    - |
+      set -e
+      ls | tee $0
+    - {outputPath: downloaded}
+"""
+
+OP2_STR = """
+name: echo
+inputs:
+  - {name: msg, type: String}
+implementation:
+  container:
+    image: library/bash
+    command:
+    - sh
+    - -c
+    args:
+    - |
+      set -e
+      echo
+    - {inputValue: msg}
+"""
+
+op1_op = components.load_component_from_text(OP1_STR)
+op2_op = components.load_component_from_text(OP2_STR)
 
 
 @dsl.pipeline(
-    name='Volume',
+    name='volume',
     description='A pipeline with volume.'
 )
 def volume_pipeline():
-    op1 = dsl.ContainerOp(name='download',
-                          image='google/cloud-sdk',
-                          command=['sh', '-c'],
-                          arguments=['ls | tee /tmp/results.txt'],
-                          file_outputs={'downloaded': '/tmp/results.txt'})
+    op1 = op1_op()
+
     op1.add_volume(V1Volume(name='gcp-credentials',
                             secret=V1SecretVolumeSource(secret_name='user-gcp-sa')))
     op1.container.add_volume_mount(V1VolumeMount(mount_path='/secret/gcp-credentials',
@@ -35,10 +70,7 @@ def volume_pipeline():
                                             value='/secret/gcp-credentials/user-gcp-sa.json'))
     op1.container.add_env_variable(V1EnvVar(name='Foo', value='bar'))
 
-    op2 = dsl.ContainerOp(name='echo',
-                          image='library/bash',
-                          command=['sh', '-c'],
-                          arguments=['echo %s' % op1.output])
+    op2 = op2_op(op1.output)
 
 
 if __name__ == '__main__':

diff --git a/sdk/python/tests/compiler/testdata/volume.yaml b/sdk/python/tests/compiler/testdata/volume.yaml
@@ -18,7 +18,7 @@ metadata:
   name: volume
   annotations:
     tekton.dev/output_artifacts: '{"download": [{"key": "artifacts/$PIPELINERUN/download/downloaded.tgz",
-      "name": "download-downloaded", "path": "/tmp/results.txt"}]}'
+      "name": "download-downloaded", "path": "/tmp/outputs/downloaded/data"}]}'
     tekton.dev/input_artifacts: '{"echo": [{"name": "download-downloaded", "parent_task":
       "download"}]}'
     tekton.dev/artifact_bucket: mlpipeline
@@ -28,7 +28,7 @@ metadata:
       "echo": []}'
     sidecar.istio.io/inject: "false"
     pipelines.kubeflow.org/pipeline_spec: '{"description": "A pipeline with volume.",
-      "name": "Volume"}'
+      "name": "volume"}'
 spec:
   pipelineSpec:
     tasks:
@@ -37,7 +37,10 @@ spec:
         steps:
         - name: main
           args:
-          - ls | tee $(results.downloaded.path)
+          - |
+            set -e
+            ls | tee $0
+          - $(results.downloaded.path)
           command:
           - sh
           - -c
@@ -52,7 +55,7 @@ spec:
             name: gcp-credentials
         results:
         - name: downloaded
-          description: /tmp/results.txt
+          description: /tmp/outputs/downloaded/data
         volumes:
         - name: gcp-credentials
           secret:
@@ -63,6 +66,10 @@ spec:
             pipelines.kubeflow.org/generation: ''
             pipelines.kubeflow.org/cache_enabled: "true"
           annotations:
+            pipelines.kubeflow.org/component_spec: '{"implementation": {"container":
+              {"args": ["set -e\nls | tee $0\n", {"outputPath": "downloaded"}], "command":
+              ["sh", "-c"], "image": "google/cloud-sdk"}}, "name": "download", "outputs":
+              [{"name": "downloaded", "type": "String"}]}'
             tekton.dev/template: ''
       timeout: 0s
     - name: echo
@@ -73,7 +80,10 @@ spec:
         steps:
         - name: main
           args:
-          - echo $(inputs.params.download-downloaded)
+          - |
+            set -e
+            echo
+          - $(inputs.params.download-downloaded)
           command:
           - sh
           - -c
@@ -86,6 +96,10 @@ spec:
             pipelines.kubeflow.org/generation: ''
             pipelines.kubeflow.org/cache_enabled: "true"
           annotations:
+            pipelines.kubeflow.org/component_spec: '{"implementation": {"container":
+              {"args": ["set -e\necho\n", {"inputValue": "msg"}], "command": ["sh",
+              "-c"], "image": "library/bash"}}, "inputs": [{"name": "msg", "type":
+              "String"}], "name": "echo"}'
             tekton.dev/template: ''
       timeout: 0s
   timeout: 0s
diff --git a/sdk/python/tests/compiler/testdata/volume_noninlined.yaml b/sdk/python/tests/compiler/testdata/volume_noninlined.yaml
@@ -18,7 +18,7 @@ metadata:
   name: volume
   annotations:
     tekton.dev/output_artifacts: '{"download": [{"key": "artifacts/$PIPELINERUN/download/downloaded.tgz",
-      "name": "download-downloaded", "path": "/tmp/results.txt"}]}'
+      "name": "download-downloaded", "path": "/tmp/outputs/downloaded/data"}]}'
     tekton.dev/input_artifacts: '{"echo": [{"name": "download-downloaded", "parent_task":
       "download"}]}'
     tekton.dev/artifact_bucket: mlpipeline
@@ -28,7 +28,7 @@ metadata:
       "echo": []}'
     sidecar.istio.io/inject: "false"
     pipelines.kubeflow.org/pipeline_spec: '{"description": "A pipeline with volume.",
-      "name": "Volume"}'
+      "name": "volume"}'
 spec:
   pipelineSpec:
     tasks:
@@ -37,7 +37,10 @@ spec:
         steps:
         - name: main
           args:
-          - ls | tee $(results.downloaded.path)
+          - |
+            set -e
+            ls | tee $0
+          - $(results.downloaded.path)
           command:
           - sh
           - -c
@@ -52,7 +55,7 @@ spec:
             name: gcp-credentials
         results:
         - name: downloaded
-          description: /tmp/results.txt
+          description: /tmp/outputs/downloaded/data
         volumes:
         - name: gcp-credentials
           secret:
@@ -63,6 +66,10 @@ spec:
             pipelines.kubeflow.org/generation: ''
             pipelines.kubeflow.org/cache_enabled: "true"
           annotations:
+            pipelines.kubeflow.org/component_spec: '{"implementation": {"container":
+              {"args": ["set -e\nls | tee $0\n", {"outputPath": "downloaded"}], "command":
+              ["sh", "-c"], "image": "google/cloud-sdk"}}, "name": "download", "outputs":
+              [{"name": "downloaded", "type": "String"}]}'
             tekton.dev/template: ''
       timeout: 0s
     - name: echo
@@ -73,7 +80,10 @@ spec:
         steps:
         - name: main
           args:
-          - echo $(inputs.params.download-downloaded)
+          - |
+            set -e
+            echo
+          - $(inputs.params.download-downloaded)
           command:
           - sh
           - -c
@@ -86,6 +96,10 @@ spec:
             pipelines.kubeflow.org/generation: ''
             pipelines.kubeflow.org/cache_enabled: "true"
           annotations:
+            pipelines.kubeflow.org/component_spec: '{"implementation": {"container":
+              {"args": ["set -e\necho\n", {"inputValue": "msg"}], "command": ["sh",
+              "-c"], "image": "library/bash"}}, "inputs": [{"name": "msg", "type":
+              "String"}], "name": "echo"}'
             tekton.dev/template: ''
       timeout: 0s
   timeout: 0s
diff --git a/sdk/python/tests/compiler/testdata/volume_op.py b/sdk/python/tests/compiler/testdata/volume_op.py
@@ -13,13 +13,28 @@
 # limitations under the License.
 
 import kfp.dsl as dsl
+from kfp import components
+
+COP_STR = """
+name: cop
+implementation:
+  container:
+    image: library/bash:4.4.23
+    command:
+    - sh
+    - -c
+    args:
+    - echo foo > /mnt/file1
+"""
+
+cop_op = components.load_component_from_text(COP_STR)
 
 
 @dsl.pipeline(
-    name="VolumeOp Basic",
+    name="volumeop-basic",
     description="A Basic Example on VolumeOp Usage."
 )
-def volumeop_basic(size="10M"):
+def volumeop_basic(size: str = "10M"):
     vop = dsl.VolumeOp(
         name="create-pvc",
         resource_name="my-pvc",
@@ -29,13 +44,7 @@ def volumeop_basic(size="10M"):
         # failure_condition="status.phase = Failed"
     )
 
-    cop = dsl.ContainerOp(
-        name="cop",
-        image="library/bash:4.4.23",
-        command=["sh", "-c"],
-        arguments=["echo foo > /mnt/file1"],
-        pvolumes={"/mnt": vop.volume}
-    )
+    cop = cop_op().add_pvolumes({"/mnt": vop.volume})
 
 
 if __name__ == '__main__':

diff --git a/sdk/python/tests/compiler/testdata/volume_op.yaml b/sdk/python/tests/compiler/testdata/volume_op.yaml
@@ -26,8 +26,8 @@ metadata:
     tekton.dev/artifact_items: '{"cop": [], "create-pvc": []}'
     sidecar.istio.io/inject: "false"
     pipelines.kubeflow.org/pipeline_spec: '{"description": "A Basic Example on VolumeOp
-      Usage.", "inputs": [{"default": "10M", "name": "size", "optional": true}], "name":
-      "VolumeOp Basic"}'
+      Usage.", "inputs": [{"default": "10M", "name": "size", "optional": true, "type":
+      "String"}], "name": "volumeop-basic"}'
 spec:
   params:
   - name: size
@@ -155,6 +155,9 @@ spec:
             pipelines.kubeflow.org/generation: ''
             pipelines.kubeflow.org/cache_enabled: "true"
           annotations:
+            pipelines.kubeflow.org/component_spec: '{"implementation": {"container":
+              {"args": ["echo foo > /mnt/file1"], "command": ["sh", "-c"], "image":
+              "library/bash:4.4.23"}}, "name": "cop"}'
             tekton.dev/template: ''
       timeout: 0s
   timeout: 0s
diff --git a/sdk/python/tests/compiler/testdata/volume_op_noninlined.yaml b/sdk/python/tests/compiler/testdata/volume_op_noninlined.yaml
@@ -26,8 +26,8 @@ metadata:
     tekton.dev/artifact_items: '{"cop": [], "create-pvc": []}'
     sidecar.istio.io/inject: "false"
     pipelines.kubeflow.org/pipeline_spec: '{"description": "A Basic Example on VolumeOp
-      Usage.", "inputs": [{"default": "10M", "name": "size", "optional": true}], "name":
-      "VolumeOp Basic"}'
+      Usage.", "inputs": [{"default": "10M", "name": "size", "optional": true, "type":
+      "String"}], "name": "volumeop-basic"}'
 spec:
   params:
   - name: size
@@ -155,6 +155,9 @@ spec:
             pipelines.kubeflow.org/generation: ''
             pipelines.kubeflow.org/cache_enabled: "true"
           annotations:
+            pipelines.kubeflow.org/component_spec: '{"implementation": {"container":
+              {"args": ["echo foo > /mnt/file1"], "command": ["sh", "-c"], "image":
+              "library/bash:4.4.23"}}, "name": "cop"}'
             tekton.dev/template: ''
       timeout: 0s
   timeout: 0s
diff --git a/sdk/python/tests/compiler/testdata/volume_snapshot_op.log b/sdk/python/tests/compiler/testdata/volume_snapshot_op.log