fix(components): Update GCP component container to Python 3.7. Fixes #4959

[chensun]

Description of your changes:

• Update GCP component container to using Python 3.7, also upgrade apache-beam[gcp] version.
• Fix Dataflow client and sample.

Tested via Dataflow launch_python and launch_template sample notebooks.
Fixes #4959

Checklist:

• The title for your pull request (PR) should follow our title convention. Learn more about the pull request title convention used in this repository.

• Do you want this pull request (PR) cherry-picked into the current release branch?

  Learn more about cherry-picking updates into the release branch.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[neuromage]

/lgtm

[Ark-kun]

Thank you for tackling this, Chen.

A general comment: It might be beneficial to split unrelated changes into separate PRs. It looks like this PR changes several things at once - Python 2, API regionalization and general refactoring.

[chensun]

Thank you for tackling this, Chen.

A general comment: It might be beneficial to split unrelated changes into separate PRs. It looks like this PR changes several things at once - Python 2, API regionalization and general refactoring.

All changes are actually related. The apache-beam[gcp] version we pinned previously only works for Python 2.7. So as we upgrade to Python 3.7, we need to use a newer version of apache-beam[gcp]. And I found out that new SDK requires a few changes:

• runner needs to be "DatatflowRunner" instead of just "dataflow"
• region is now a required argument, otherwise we got compile time error: 'ValueError: Pipeline has validations errors: Missing required option: region.'

Also the backend now returns job URL in a different format, so we need to change the parsing logic. Though this is not a sdk/container related change, but it affects e2e runtime results which is currently broken.

[chensun]

/test kubeflow-pipeline-frontend-test

[numerology]

/lgtm
/approve

Thanks @chensun !

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: chensun, numerology

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• components/OWNERS [numerology]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment