Feature branch for moving to Tekton 0.16 (kubeflow#321)

* add tekton 0.16.3 go mod

* update dev setup

* update dev docs

* update readme

* support more argo variable mapping

* update feature doc on the list of supported argo variables translation

* add task-wise metadata support

* add beta whenexpression for condition, unblock runAfter for conditional task

* update condition docs

* update ui image for tekton-0.16

* Apply suggestions from code review

Co-authored-by: Andrew Butler <Andrew.Butler@ibm.com>

* address comments and fix conflicts

* update doc reference to tekton 0.16

* regenerate tests

* updated latest images with tekton 0.16

* regenerate tests

Co-authored-by: Andrew Butler <Andrew.Butler@ibm.com>

README.md

@@ -17,7 +17,7 @@ has Steps. Please look for more details in [Tekton repo](https://github.com/tekt
 ## Kubeflow Pipeline with Tekton Backend
 
 We are currently using [Kubeflow Pipelines 1.0.4](https://github.com/kubeflow/pipelines/releases/tag/1.0.4) and
-[Tekton >= 0.14.0](https://github.com/tektoncd/pipeline/releases/tag/v0.14.0) for this project.
+[Tekton >= 0.16.0](https://github.com/tektoncd/pipeline/releases/tag/v0.16.0) for this project.
 
 ![kfp-tekton](images/kfp-tekton.png)
 

go.mod

@@ -31,7 +31,7 @@ require (
 	github.com/sirupsen/logrus v1.6.0
 	github.com/spf13/viper v1.7.0
 	github.com/stretchr/testify v1.5.1
-	github.com/tektoncd/pipeline v0.15.0
+	github.com/tektoncd/pipeline v0.16.3
 	github.com/valyala/fasttemplate v1.1.0 // indirect
 	golang.org/x/net v0.0.0-20200520182314-0ba52f642ac2
 	golang.org/x/sync v0.0.0-20201008141435-b3e1573b7520 // indirect

guides/developer_guide.md

@@ -28,8 +28,7 @@ If you would like to do it in development mode, or if you already have a Kubeflo
 ## Prerequisites
 
 1. [Install Tekton](https://github.com/tektoncd/pipeline/blob/master/docs/install.md#installing-tekton-pipelines-on-kubernetes).
-    - Minimum version: `0.14.0`
-    - Recommended version: `0.15.0`
+    - Minimum version: `0.16.0`
 2. Clone this repository
     ```
     git clone github.com/kubeflow/kfp-tekton

guides/kfp_tekton_install.md

@@ -35,7 +35,7 @@ A Kubernetes cluster `v1.16` that has least 8 vCPU and 16 GB memory.
 
 ## Standalone Kubeflow Pipelines with Tekton Backend Deployment
 To install the standalone Kubeflow Pipelines with Tekton, run the following steps:
-1. Install [Tekton v0.14.3](https://github.com/tektoncd/pipeline/releases/tag/v0.14.3)
+1. Install [Tekton v0.16.3](https://github.com/tektoncd/pipeline/releases/tag/v0.16.3)
 
 2. Install Kubeflow Pipelines with Tekton backend (kfp-tekton) v0.4.0 release
     ```shell

samples/README.md

@@ -1,13 +1,13 @@
 # KFP Tekton Samples
 
-Below are the list of samples that are currently running end to end taking the compiled Tekton yaml and deploying on a Tekton cluster directly. 
+Below are the list of samples that are currently running end to end taking the compiled Tekton yaml and deploying on a Tekton cluster directly.
 If you are interested more in the larger list of pipelines samples we are testing for whether they can be 'compiled to Tekton' format, please [look at the corresponding status page](/sdk/python/tests/README.md)
 
 [KFP Tekton User Guide](/guides/kfp-user-guide) is a guideline for the possible ways to develop and consume Kubeflow Pipeline with Tekton. It's recommended to go over at least one of the methods in the user guide before heading into the KFP Tekton Samples.
 
-## Prerequisites 
+## Prerequisites
 - Install [Kubeflow 1.0.2+](https://www.kubeflow.org/docs/started/getting-started/) and connect the cluster to the current shell with `kubectl`
-- Install [Tekton 0.14.0](https://github.com/tektoncd/pipeline/releases/tag/v0.14.0)
+- Install [Tekton 0.16.3](https://github.com/tektoncd/pipeline/releases/tag/v0.16.3)
     - For KFP, we shouldn't be modifying the default work directory for any component. Therefore, please run the below command to disable the [home and work directories overwrite](https://github.com/tektoncd/pipeline/blob/master/docs/install.md#customizing-the-pipelines-controller-behavior) from Tekton default.
         ```shell
         kubectl patch cm feature-flags -n tekton-pipelines -p '{"data":{"disable-home-env-overwrite":"true","disable-working-directory-overwrite":"true"}}'

sdk/FEATURES.md

@@ -119,7 +119,7 @@ This feature has been available since Tekton version `0.13.0`.
 
 An _exit handler_ is a component that always executes, irrespective of success or failure,
 at the end of the pipeline. It is implemented using Tekton's
-[finally](https://github.com/tektoncd/pipeline/blob/v0.14.0/docs/pipelines.md#adding-finally-to-the-pipeline)
+[finally](https://github.com/tektoncd/pipeline/blob/v0.16.0/docs/pipelines.md#adding-finally-to-the-pipeline)
 section under the Pipeline `spec`. An example of how to use an _exit handler_ can be found in
 the [exit_handler](/sdk/python/tests/compiler/testdata/exit_handler.py) compiler test.
 
@@ -142,12 +142,9 @@ is an example of how to use this feature.
 
 ### Conditions
 
-Conditions are for determining whether to execute certain components based on the output of the condition checks. Since Tekton required users to define an image for doing the [condition check](https://github.com/tektoncd/pipeline/blob/master/docs/conditions.md), we created a custom python image to replicate the same condition checks from Argo and made it as the default in our compiler. The
+Conditions are used for determining whether to execute certain components based on the output of the condition checks. In KFP Argo, each condition is represented as an Argo DAG template so it can be used as a dependency for other Argo templates. To replicate this in KFP Tekton, we put our condition into a dedicated Tekton task so that conditions can be treated as a dependency for other Tekton tasks. Another advantage of creating conditions using Tekton tasks is that we can have more flexible conditions such as comparing an integer and a float number, which currently is not available in Tekton. We are using the Tekton [when expression](https://github.com/tektoncd/pipeline/blob/master/docs/pipelines.md#guard-task-execution-using-whenexpressions) to check whether the condition task has succeeded or not. We created a custom python image to replicate the same condition checks that are in Argo and made it as the default in our compiler. The
 [flip-coin](/samples/flip-coin) example demonstrates how to use multiple conditions within the same pipeline.
 
-Please be aware that the current Condition feature is using Tekton V1alpha1 API because the Tekton community is still designing the V1beta1 API.
-We will be migrating to the V1beta1 API once it's available in Tekton. Please refer to the [design proposal](https://docs.google.com/document/d/1kESrgmFHnirKNS4oDq3mucuB_OycBm6dSCSwRUHccZg/edit?usp=sharing) for more details.
-
 ### ResourceOp, VolumeOp, and VolumeSnapshotOp
 
 [ResourceOp, VolumeOp, and VolumeSnapshotOp](https://www.kubeflow.org/docs/pipelines/sdk/manipulate-resources/) are special operations for
@@ -206,6 +203,10 @@ However, when using dynamic parameters, the number of parallel tasks is determin
 argo -> tekton
 {{inputs.parameters.%s}} -> $(inputs.params.%s)
 {{outputs.parameters.%s}} -> $(results.%s.path)
+{{workflow.uid}} -> $(context.pipelineRun.uid)
+{{workflow.name}} -> $(context.pipelineRun.name)
+{{workflow.namespace}} -> $(context.pipelineRun.namespace)
+{{workflow.parameters.%s}} -> $(params.%s)
 ```
 
 [parallel_join_with_argo_vars](/sdk/python/tests/compiler/testdata/parallel_join_with_argo_vars.py) is an example of how Argo variables are

sdk/README.md

@@ -36,7 +36,7 @@ adding the `TektonCompiler` and the `TektonClient`:
 
   * `kfp_tekton.compiler.TektonCompiler.compile` compiles your Python DSL code
     into a single static configuration (in YAML format) that the Kubeflow Pipelines service
-    can process. The Kubeflow Pipelines service converts the static 
+    can process. The Kubeflow Pipelines service converts the static
     configuration into a set of Kubernetes resources for execution.
 
 * `kfp_tekton.TektonClient` contains the Python client libraries for the [Kubeflow Pipelines API](https://www.kubeflow.org/docs/pipelines/reference/api/kubeflow-pipeline-api-spec/).
@@ -49,14 +49,14 @@ adding the `TektonCompiler` and the `TektonClient`:
   * `kfp_tekton.TektonClient.run_pipeline` runs a pipeline and returns a run object.
   * `kfp_tekton.TektonClient.create_run_from_pipeline_func` compiles a pipeline
     function and submits it for execution on Kubeflow Pipelines.
-  * `kfp_tekton.TektonClient.create_run_from_pipeline_package` runs a local 
+  * `kfp_tekton.TektonClient.create_run_from_pipeline_package` runs a local
     pipeline package on Kubeflow Pipelines.
 
 
 ## Project Prerequisites
 
  - Python: `3.5.3` or later
- - Tekton: [`v0.14.0`](https://github.com/tektoncd/pipeline/releases/tag/v0.14.0) or [later](https://github.com/tektoncd/pipeline/releases/latest)
+ - Tekton: [`v0.16.3`](https://github.com/tektoncd/pipeline/releases/tag/v0.16.3) or [later](https://github.com/tektoncd/pipeline/releases/latest)
  - Tekton CLI: [`0.11.0`](https://github.com/tektoncd/cli/releases/tag/v0.11.0)
  - Kubeflow Pipelines: [KFP with Tekton backend](/guides/kfp_tekton_install.md)
 
@@ -72,9 +72,9 @@ virtual environment first:
 
     python3 -m venv .venv
     source .venv/bin/activate
-    
+
     pip install kfp-tekton
-    
+
 Alternatively you can install the latest version of the `kfp-tekton` compiler
 from source by cloning the repository [https://github.com/kubeflow/kfp-tekton](https://github.com/kubeflow/kfp-tekton):
 
@@ -159,19 +159,19 @@ You can run the pipeline directly using a pre-compiled file and KFP-Tekton SDK.
 ```python
 experiment = kfp_tekton.TektonClient.create_experiment(name=EXPERIMENT_NAME, namespace=KUBEFLOW_PROFILE_NAME)
 run = client.run_pipeline(experiment.id, 'parallal-join-pipeline', 'pipeline.yaml')
-``` 
+```
 
 You can also deploy directly on Tekton cluster with `kubectl`. The Tekton server will automatically start a pipeline run.
 We can then follow the logs using the `tkn` CLI.
 
     kubectl apply -f pipeline.yaml
-    
+
     tkn pipelinerun logs --last --follow
 
 Once the Tekton Pipeline is running, the logs should start streaming:
-      
+
     Waiting for logs to be available...
-    
+
     [gcs-download : main] With which he yoketh your rebellious necks Razeth your cities and subverts your towns And in a moment makes them desolate
 
     [gcs-download-2 : main] I find thou art no less than fame hath bruited And more than may be gatherd by thy shape Let my presumption not provoke thy wrath
@@ -204,7 +204,7 @@ your code changes are improving the number of successfully compiled KFP pipeline
 
 - When you encounter ServiceAccount related permission issues, refer to the
   ["Service Account and RBAC" doc](sa-and-rbac.md)
-  
-- If you run into the error `bad interpreter: No such file or director` when trying 
+
+- If you run into the error `bad interpreter: No such file or director` when trying
   to use Python's venv, remove the current virtual environment in the `.venv` directory
   and create a new one using `virtualenv .venv`

sdk/python/README.md

@@ -1,6 +1,6 @@
 # KFP-Tekton Developer Guide
 
-This document describes the development guidelines for contributing to the KFP-Tekton project. 
+This document describes the development guidelines for contributing to the KFP-Tekton project.
 Details about the required contributor license agreement (CLA) and the code review process can be found
 in the [CONTRIBUTING.md](/CONTRIBUTING.md) document.
 A quick-start guide with general setup instruction, trouble shooting guide and technical limitations
@@ -36,7 +36,7 @@ can be found in the [SDK README](/sdk/README.md)
 1. [`Python`](https://www.python.org/downloads/): version `3.5.3` or later (new code must maintain compatibility with `3.5`)
 2. [`Kubernetes` Cluster](https://v1-15.docs.kubernetes.io/docs/setup/): version `1.15` ([required by Kubeflow](https://www.kubeflow.org/docs/started/k8s/overview/) and Tekton 0.11)
 3. [`kubectl` CLI](https://kubernetes.io/docs/tasks/tools/install-kubectl/): required to deploy Tekton pipelines to Kubernetes cluster
-4. [`Tekton` Deployment](https://github.com/tektoncd/pipeline/releases/tag/v0.15.0/): version `0.14.0` or greater (minimum version `0.13.0` to support Tekton API version `v1beta1`), required for end-to-end testing
+4. [`Tekton` Deployment](https://github.com/tektoncd/pipeline/releases/tag/v0.15.0/): version `0.16.3` or greater, required for end-to-end testing
 5. [`tkn` CLI](https://github.com/tektoncd/cli#installing-tkn): version `0.11.0` or greater, required for end-to-end testing of Tekton pipelines
 6. [`Kubeflow Pipelines` Deployment](https://www.kubeflow.org/docs/pipelines/installation/overview/): required for some end-to-end tests
 
@@ -74,7 +74,7 @@ branch. Currently there are no features that require a special build.
 
 #### Tekton CLI
 
-Follow the instructions [here](https://github.com/tektoncd/cli#installing-tkn). 
+Follow the instructions [here](https://github.com/tektoncd/cli#installing-tkn).
 
 Mac OS users can install the Tekton CLI using the `homebrew` formula:
 
@@ -91,7 +91,7 @@ The Tekton Dashboard can be accessed through its `ClusterIP` service by running
 be patched to expose a public `NodePort` IP:
 
   	kubectl patch svc tekton-dashboard -n tekton-pipelines --type='json' -p '[{"op":"replace","path":"/spec/type","value":"NodePort"}]'
-   
+
 To open the dashboard run:
 
     TKN_DASHBOARD_SVC_PORT=$(kubectl -n tekton-pipelines get service tekton-dashboard -o jsonpath='{.spec.ports[0].nodePort}')
@@ -107,8 +107,8 @@ like the DSL and components packages, but "override" or "replace" those parts of
 the Tekton YAML instead of Argo YAML. Since the KFP SDK was not designed and implemented to easily be extended,
 _monkey-patching_ was used to replace non-class methods and functions at runtime.
 
-In order for the _monkey patch_ to work properly, the `kfp-tekton` compiler source code has to be aligned with a 
-specific version of the `kfp` SDK compiler. As of now that version is [`1.0.4`](https://github.com/kubeflow/pipelines/releases/tag/1.0.4). 
+In order for the _monkey patch_ to work properly, the `kfp-tekton` compiler source code has to be aligned with a
+specific version of the `kfp` SDK compiler. As of now that version is [`1.0.4`](https://github.com/kubeflow/pipelines/releases/tag/1.0.4).
 
 
 ## Adding New Code
@@ -118,12 +118,12 @@ The Python package structure as well as the module names and method signatures c
 This helps keeping track of all the code that had to modified and will make merging (some of) the code back into KFP or
 identify pieces of code that need to be refactored in KFP in order to accommodate various execution platforms.
 When it is necessary to bring further methods from `kfp` compiler package into the `kfp-tekton` compiler package, keep
-the original method names and signatures as well as their position inside their respective Python modules. 
+the original method names and signatures as well as their position inside their respective Python modules.
 
 Be sure to run `make verify` before committing your code changes and creating a pull request:
 
     $ make verify
-    
+
     check_license: OK
     lint: OK
     unit_test: OK
@@ -138,7 +138,7 @@ Most of the functions provided by the `kfp.compiler.compiler.Compiler` are insta
 
 Static `Compiler` methods may need to be also be added to the _monkey patch_ described in the next
 section unless they are only used by other methods that are already overridden in `TektonCompiler`.
-Be careful not to mix inheritance and monkey patching. A method which in its body calls on its 
+Be careful not to mix inheritance and monkey patching. A method which in its body calls on its
 `super().` implementation must not be added to the list of methods that get dynamically replaced
 via the _monkey patch_.
 
@@ -149,8 +149,8 @@ should be added to their respective modules in `kfp_tekton.compiler` and added t
 which dynamically replaces the code in the `kfp` at runtime.
 
 As of May 2020, the _monkey patch_ was no longer needed and removed since all of the _patched_ methods
-are now invoked directly (and exclusively) by other code implemented in the `kfp_tekton` compiler. 
-Details on how to implement a _monkey patch_ can be found in the 
+are now invoked directly (and exclusively) by other code implemented in the `kfp_tekton` compiler.
+Details on how to implement a _monkey patch_ can be found in the
 [Removed Features](#monkey-patch-to-dynamically-replace-static-kfp-compiler-methods)
 section if it becomes necessary to reintroduce the _monkey patch_ for any methods we need to "override"
 which are not exclusively called directly by other methods we already implemented in the `kfp_tekton`
@@ -190,7 +190,7 @@ run on a Tekton cluster.
 Any new functionality being added to the `kfp_tekton.compiler` should be accompanied by a new unit test in `sdk/python/tests/compiler/compiler_tests.py`
 Typically a test case comes with a minimal Python DSL script and a "golden" YAML file in `sdk/python/tests/compiler/testdata`.
 The "golden" YAML file contains the expected compiler output. The unit tests use the "golden" YAML files to compare
-the current compiler output with the previously expected compiler output. 
+the current compiler output with the previously expected compiler output.
 
     make unit_test
 
@@ -207,7 +207,7 @@ environment variable:
 
 The unit tests are designed to verify the YAML produced by the compiler matches the expected, previously generated
 "golden" YAML. End-to-end (E2E) tests are necessary to verify that the generated Tekton YAML is syntactically valid and
-that the pipeline can be executed successfully on a Tekton cluster. 
+that the pipeline can be executed successfully on a Tekton cluster.
 
 A manual E2E test can be performed in the following manner:
 
@@ -222,7 +222,7 @@ access secrets:
     tkn pipeline start <pipeline-name> --showlog -n kubeflow
 
 You can also run the dynamically generated end-to-end test suite which takes all of the "golden" YAML files from the
-compiler `testdata` directory and runs them on a Kubernetes cluster, prerequisite that the environment variable 
+compiler `testdata` directory and runs them on a Kubernetes cluster, prerequisite that the environment variable
 `KUBECONFIG` is set and the K8s cluster has both Kubeflow Pipelines as well as Tekton Pipelines installed:
 
     make e2e_test
@@ -319,7 +319,7 @@ except Exception as error:
 ```
 
 **Note**: Since the _monkey patch_ gets triggered by importing any member of the `kfp_tekton.compiler` module, we try to
-avoid using top-level imports of any members in `kfp_tekton.compiler` in pipeline DSL scripts. 
+avoid using top-level imports of any members in `kfp_tekton.compiler` in pipeline DSL scripts.
 Instead use local imports to avoid triggering the _monkey-patch_ when the original KFP compiler is used to compile a
 pipeline DSL script using KFP's `dsl-compile --py <DSL script>` command.