Upgrade Python SDK to kfp==1.6.3 (kubeflow#617)

Resolves kubeflow#595

Signed-off-by: Christian Kadner <ckadner@us.ibm.com>

sdk/python/README.md

@@ -108,7 +108,8 @@ the Tekton YAML instead of Argo YAML. Since the KFP SDK was not designed and imp
 _monkey-patching_ was used to replace non-class methods and functions at runtime.
 
 In order for the _monkey patch_ to work properly, the `kfp-tekton` compiler source code has to be aligned with a
-specific version of the `kfp` SDK compiler. As of now that version is [`1.5.0`](https://github.com/kubeflow/pipelines/releases/tag/1.5.0).
+specific version of the `kfp` SDK compiler. As of now the `kfp-tekton` SDK version is `0.8.1` which is aligned with KFP 
+SDK version [`1.6.3`](https://pypi.org/project/kfp/1.6.3/).
 
 
 ## Adding New Code
@@ -223,7 +224,7 @@ access secrets:
 
 You can also run the dynamically generated end-to-end test suite which takes all of the "golden" YAML files from the
 compiler `testdata` directory and runs them on a Kubernetes cluster, prerequisite that the environment variable
-`KUBECONFIG` is set and the K8s cluster has both Kubeflow Pipelines as well as Tekton Pipelines installed:
+`KUBECONFIG` is set and the K8s cluster has both Kubeflow Pipelines and Tekton Pipelines installed:
 
     make e2e_test
 
@@ -246,7 +247,7 @@ To update the ["Compiler Status Report"](/sdk/python/tests/README.md) use the ou
 ## License Headers
 
 All source files should have the following license header. Adjust the year accordingly to reflect the year the file was
-added and the last year it was modified:
+added, and the last year it was modified:
 
     # Copyright 2019-2021 kubeflow.org
     #
@@ -329,6 +330,3 @@ if __name__ == '__main__':
     from kfp_tekton.compiler import TektonCompiler
     TektonCompiler().compile(pipeline_func, __file__.replace('.py', '.yaml'))
 ```
-
-
-

sdk/python/kfp_tekton/__init__.py

@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-__version__ = '0.8.0'
+__version__ = '0.8.1'
 
 from ._client import TektonClient  # noqa F401
 from .k8s_client_helper import env_from_secret  # noqa F401

sdk/python/requirements.in

@@ -1,3 +1 @@
-# TODO
-#kfp==1.5.0
-git+https://github.com/kubeflow/pipelines.git@1.5.0#egg=kfp&subdirectory=sdk/python
+kfp==1.6.3

sdk/python/requirements.txt

@@ -30,10 +30,18 @@ docstring-parser==0.7.3
 fire==0.4.0
     # via kfp
 google-api-core==1.26.0
-    # via google-cloud-core
+    # via
+    #   google-api-python-client
+    #   google-cloud-core
+google-api-python-client==1.12.8
+    # via kfp
+google-auth-httplib2==0.1.0
+    # via google-api-python-client
 google-auth==1.27.0
     # via
     #   google-api-core
+    #   google-api-python-client
+    #   google-auth-httplib2
     #   google-cloud-core
     #   google-cloud-storage
     #   kfp
@@ -48,17 +56,19 @@ google-resumable-media==1.2.0
     # via google-cloud-storage
 googleapis-common-protos==1.52.0
     # via google-api-core
+httplib2==0.19.1
+    # via
+    #   google-api-python-client
+    #   google-auth-httplib2
 idna==2.10
     # via requests
 jsonschema==3.2.0
     # via kfp
-kfp-pipeline-spec==0.1.6
+kfp-pipeline-spec==0.1.8
     # via kfp
 kfp-server-api==1.3.0
     # via kfp
-# kfp==1.5.0
-    # TODO
-git+https://github.com/kubeflow/pipelines.git@1.5.0#egg=kfp&subdirectory=sdk/python
+kfp==1.6.3
     # via -r sdk/python/requirements.in
 kubernetes==12.0.1
     # via kfp
@@ -70,6 +80,7 @@ protobuf==3.15.2
     # via
     #   google-api-core
     #   googleapis-common-protos
+    #   kfp
     #   kfp-pipeline-spec
 pyasn1-modules==0.2.8
     # via google-auth
@@ -80,7 +91,9 @@ pyasn1==0.4.8
 pycparser==2.20
     # via cffi
 pyparsing==2.4.7
-    # via packaging
+    # via
+    #   httplib2
+    #   packaging
 pyrsistent==0.17.3
     # via jsonschema
 python-dateutil==2.8.1
@@ -111,7 +124,9 @@ six==1.15.0
     #   absl-py
     #   fire
     #   google-api-core
+    #   google-api-python-client
     #   google-auth
+    #   google-auth-httplib2
     #   google-cloud-core
     #   google-resumable-media
     #   jsonschema
@@ -126,6 +141,8 @@ tabulate==0.8.9
     # via kfp
 termcolor==1.1.0
     # via fire
+uritemplate==3.0.1
+    # via google-api-python-client
 urllib3==1.26.3
     # via
     #   kfp-server-api

sdk/python/setup.py

@@ -20,14 +20,14 @@
 #
 # To create a distribution for PyPi run:
 #
-#    $ export KFP_TEKTON_VERSION=0.8.0-rc1
+#    $ export KFP_TEKTON_VERSION=0.8.1-rc1
 #    $ python3 setup.py sdist
 #    $ twine check dist/kfp-tekton-${KFP_TEKTON_VERSION/-rc/rc}.tar.gz
 #    $ twine upload --repository pypi dist/kfp-tekton-${KFP_TEKTON_VERSION/-rc/rc}.tar.gz
 #
 #   ... or:
 #
-#    $ make distribution KFP_TEKTON_VERSION=0.8.0-rc1
+#    $ make distribution KFP_TEKTON_VERSION=0.8.1-rc1
 #
 # =============================================================================
 
@@ -54,7 +54,7 @@
 # NOTICE, after any updates to the following, ./requirements.in should be updated
 # accordingly.
 REQUIRES = [
-    "kfp @ git+https://github.com/kubeflow/pipelines.git@1.5.0#egg=kfp&subdirectory=sdk/python",
+    "kfp==1.6.3",
 ]
 
 TESTS_REQUIRE = [

sdk/python/tests/README.md

@@ -27,7 +27,7 @@ You should see an output similar to the one below, outlining which test scripts
 which are failing:
 
 ```YAML
-KFP version: 1.5.0
+KFP SDK version: 1.6.3
 
 SUCCESS: add_pod_env.py
 SUCCESS: artifact_passing_using_volume.py
@@ -140,5 +140,5 @@ Occurences of other Errors:
 
 ## Disclaimer
 
-**Note:** The reports above were created for the pipeline scripts found in KFP version `1.5.0` since
-the `kfp_tekton` compiler code is currently based on the `kfp` SDK compiler version `1.5.0`.
+**Note:** The reports above were created for the pipeline scripts found in KFP SDK version `1.6.3` since
+the `kfp_tekton` `0.8.1` compiler code is based on the `kfp` SDK compiler version `1.6.3`.

sdk/python/tests/compiler/testdata/resourceop_basic.yaml

@@ -67,7 +67,7 @@ spec:
             is failure.
           name: failure-condition
           type: string
-        - default: aipipeline/kubectl-wrapper:0.8.0
+        - default: aipipeline/kubectl-wrapper:0.8.1
           description: Kubectl wrapper image
           name: image
           type: string

sdk/python/tests/compiler/testdata/volume_op.yaml

@@ -78,7 +78,7 @@ spec:
             is failure.
           name: failure-condition
           type: string
-        - default: aipipeline/kubectl-wrapper:0.8.0
+        - default: aipipeline/kubectl-wrapper:0.8.1
           description: Kubectl wrapper image
           name: image
           type: string

sdk/python/tests/compiler/testdata/volume_snapshot_op.yaml

@@ -84,7 +84,7 @@ spec:
             is failure.
           name: failure-condition
           type: string
-        - default: aipipeline/kubectl-wrapper:0.8.0
+        - default: aipipeline/kubectl-wrapper:0.8.1
           description: Kubectl wrapper image
           name: image
           type: string
@@ -197,7 +197,7 @@ spec:
             is failure.
           name: failure-condition
           type: string
-        - default: aipipeline/kubectl-wrapper:0.8.0
+        - default: aipipeline/kubectl-wrapper:0.8.1
           description: Kubectl wrapper image
           name: image
           type: string
@@ -310,7 +310,7 @@ spec:
             is failure.
           name: failure-condition
           type: string
-        - default: aipipeline/kubectl-wrapper:0.8.0
+        - default: aipipeline/kubectl-wrapper:0.8.1
           description: Kubectl wrapper image
           name: image
           type: string
@@ -423,7 +423,7 @@ spec:
             is failure.
           name: failure-condition
           type: string
-        - default: aipipeline/kubectl-wrapper:0.8.0
+        - default: aipipeline/kubectl-wrapper:0.8.1
           description: Kubectl wrapper image
           name: image
           type: string

sdk/python/tests/test_kfp_samples.sh

@@ -52,9 +52,9 @@ function help {
 # process command line parameters
 while (( $# > 0 )); do
   case "$1" in
-    -v|--kfp-version)          KFP_VERSION="$2";            shift 2 ;;  # KFP SDK version, default: 1.0.0
+    -v|--kfp-version)          KFP_VERSION="$2";            shift 2 ;;  # KFP SDK version, default: 1.5.0
     -a|--include-all-samples)  ALL_SAMPLES="TRUE";          shift 1 ;;  # Compile all DSL scripts in KFP repo
-    -s|--dont-list-files)       SKIP_FILES="TRUE";           shift 1 ;;  # Suppress compile status for each DSL file
+    -s|--dont-list-files)      SKIP_FILES="TRUE";           shift 1 ;;  # Suppress compile status for each DSL file
     -e|--print-error-details)  PRINT_ERRORS="TRUE";         shift 1 ;;  # Print summary of compilation errors
     -h|--help)                 help;                        exit 0  ;;  # Show this help message
     -*)                        echo "Unknown option '$1'";  exit 1  ;;
@@ -63,7 +63,8 @@ while (( $# > 0 )); do
 done
 
 # define global variables
-KFP_VERSION=${KFP_VERSION:-1.5.0}
+KFP_GIT_VERSION=${KFP_VERSION:-1.5.0}
+KFP_SDK_VERSION=${KFP_VERSION:-1.6.3}
 KFP_REPO_URL="https://github.com/kubeflow/pipelines.git"
 SCRIPT_DIR="$(cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd)"
 PROJECT_DIR="${TRAVIS_BUILD_DIR:-$(cd "${SCRIPT_DIR%/sdk/python/tests}"; pwd)}"
@@ -76,6 +77,10 @@ COMPILER_OUTPUTS_FILE="${TEMP_DIR}/test_kfp_samples_output.txt"
 CONFIG_FILE="${PROJECT_DIR}/sdk/python/tests/config.yaml"
 REPLACE_EXCEPTIONS="FALSE" # "TRUE" | "FALSE"
 
+# print KFP SDK and GIT versions (might be different)
+#echo "KFP_GIT_VERSION=${KFP_GIT_VERSION}"
+#echo "KFP_SDK_VERSION=${KFP_SDK_VERSION}"
+
 # show value of the _DIR variables, use set in POSIX mode in a sub-shell to supress functions definitions (https://stackoverflow.com/a/1305273/5601796)
 #(set -o posix ; set) | grep "_DIR" | sort
 
@@ -99,14 +104,14 @@ cp "${COMPILE_REPORT_FILE}" "${COMPILE_REPORT_FILE_OLD}"
 
 # clone kubeflow/pipeline repo to get the testdata DSL scripts
 if [ ! -d "${KFP_CLONE_DIR}" ]; then
-  git -c advice.detachedHead=false clone -b "${KFP_VERSION}" "${KFP_REPO_URL}" "${KFP_CLONE_DIR}" -q
+  git -c advice.detachedHead=false clone -b "${KFP_GIT_VERSION}" "${KFP_REPO_URL}" "${KFP_CLONE_DIR}" -q
 else
   cd "${KFP_CLONE_DIR}"
   git fetch --all -q
-  git -c advice.detachedHead=false checkout "${KFP_VERSION}" -f -q
+  git -c advice.detachedHead=false checkout "${KFP_GIT_VERSION}" -f -q
   cd - &> /dev/null
 fi
-echo "KFP version: $(git --git-dir "${KFP_CLONE_DIR}"/.git tag --points-at HEAD)"
+echo "KFP clone version: $(git --git-dir "${KFP_CLONE_DIR}"/.git tag --points-at HEAD)"
 
 # check if we are running in a Python virtual environment, if not create one
 if [ ! -d "${VENV_DIR}" ]; then
@@ -117,10 +122,10 @@ if [ ! -d "${VENV_DIR}" ]; then
 fi
 source "${VENV_DIR}/bin/activate"
 
-# install KFP with the desired KFP_VERSION (unless already installed)
-if ! (pip show "kfp" | grep Version | grep -q "${KFP_VERSION}"); then
-  echo "Installing KFP ${KFP_VERSION} ..."
-  pip install -q -e "${KFP_CLONE_DIR}/sdk/python"
+# install KFP with the desired KFP SDK version (unless already installed)
+if ! (pip show "kfp" | grep Version | grep -q "${KFP_SDK_VERSION}"); then
+  echo "Installing KFP SDK ${KFP_SDK_VERSION} ..."
+  pip install -q kfp==${KFP_SDK_VERSION}
 fi
 
 # install KFP-Tekton compiler, unless already installed
@@ -138,9 +143,13 @@ if [[ "${ALL_SAMPLES}" == "TRUE" ]]; then
 
   # reinstall KFP with the desired version to get all of its dependencies with their respective desired versions
   # pip uninstall -q -y kfp
-  pip install -q -e "${KFP_CLONE_DIR}/sdk/python"
+  pip install -q kfp==${KFP_SDK_VERSION} --force-reinstall
 fi
 
+# confirm installed kfp version(s)
+echo "KFP Python SDK version(s):"
+pip list | grep kfp
+
 echo  # just adding some separation for console output
 
 # replace NotImplementedError with simple print out