feat: add common resource paths, expose client transport (#87)

* changes without context

        autosynth cannot find the source of changes triggered by earlier changes in this
        repository, or by version upgrades to tools such as linters.

* chore(py-library): enable snippet-bot

Co-authored-by: Benjamin E. Coe <bencoe@google.com>

Source-Author: Takashi Matsuo <tmatsuo@google.com>
Source-Date: Tue Sep 1 17:14:08 2020 +0000
Source-Repo: googleapis/synthtool
Source-Sha: d91dd8aac77f7a9c5506c238038a26fa4f9e361e
Source-Link: googleapis/synthtool@d91dd8a

* chore(py-library): update decrypt secrets file

* chore(py-library): update decrypt secrets file

From https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/scripts/decrypt-secrets.sh

* docs: explain conditional

Co-authored-by: Jeffrey Rennie <rennie@google.com>

Source-Author: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com>
Source-Date: Tue Sep 8 11:35:59 2020 -0600
Source-Repo: googleapis/synthtool
Source-Sha: d302f93d7f47e2852e585ac35ab2d15585717ec0
Source-Link: googleapis/synthtool@d302f93

* chore(python-library): use sphinx 1.5.5 for the docfx job

Originally tested at:
googleapis/python-texttospeech#89

This change will fix the missing docstring in the yaml files.

Source-Author: Takashi Matsuo <tmatsuo@google.com>
Source-Date: Thu Sep 10 04:12:14 2020 +0000
Source-Repo: googleapis/synthtool
Source-Sha: ffcee7952b74f647cbb3ef021d95422f10816fca
Source-Link: googleapis/synthtool@ffcee79

* build(python): use release-publish app for notifying GitHub of releas…

* build(python): use release-publish app for notifying GitHub of release status

* fix: re-add pypi password

Source-Author: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com>
Source-Date: Wed Sep 16 08:46:42 2020 -0600
Source-Repo: googleapis/synthtool
Source-Sha: 257fda18168bedb76985024bd198ed1725485488
Source-Link: googleapis/synthtool@257fda1

* build(python): add secret manager in kokoro

Source-Author: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com>
Source-Date: Wed Sep 16 10:24:40 2020 -0600
Source-Repo: googleapis/synthtool
Source-Sha: dba48bb9bc6959c232bec9150ac6313b608fe7bd
Source-Link: googleapis/synthtool@dba48bb

* chore(python): add sphinx doctest extension

Source-Author: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com>
Source-Date: Mon Sep 21 13:09:57 2020 -0600
Source-Repo: googleapis/synthtool
Source-Sha: 27f4406999b1eee29e04b09b2423a8e4646c7e24
Source-Link: googleapis/synthtool@27f4406

* chore(python): remove note about editable installs

`pip install -e .` is supported and is how we install the library for tests.

Source-Author: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com>
Source-Date: Tue Sep 22 12:06:12 2020 -0600
Source-Repo: googleapis/synthtool
Source-Sha: a651c5fb763c69a921aecdd3e1d8dc51dbf20f8d
Source-Link: googleapis/synthtool@a651c5f

* chore(python): use BUILD_SPECIFIC_GCLOUD_PROJECT for samples

https://github.com/googleapis/python-talent/blob/ef045e8eb348db36d7a2a611e6f26b11530d273b/samples/snippets/noxfile_config.py#L27-L32

`BUILD_SPECIFIC_GCLOUD_PROJECT` is an alternate project used for sample tests that do poorly with concurrent runs on the same project.

Source-Author: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com>
Source-Date: Wed Sep 30 13:06:03 2020 -0600
Source-Repo: googleapis/synthtool
Source-Sha: 9b0da5204ab90bcc36f8cd4e5689eff1a54cc3e4
Source-Link: googleapis/synthtool@9b0da52

* chore(python): use 'setup.py' to detect repo root

Closes #792

Source-Author: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com>
Source-Date: Fri Oct 9 15:06:33 2020 -0600
Source-Repo: googleapis/synthtool
Source-Sha: e0ae456852bf22f38796deb79cff30b516fde244
Source-Link: googleapis/synthtool@e0ae456

* build(python): samples tests should pass if no samples exist

Source-Author: Daniel Sanche <d.sanche14@gmail.com>
Source-Date: Wed Oct 14 08:00:06 2020 -0700
Source-Repo: googleapis/synthtool
Source-Sha: 477764cc4ee6db346d3febef2bb1ea0abf27de52
Source-Link: googleapis/synthtool@477764c

* chore(python_library): change the docs bucket name

Source-Author: Takashi Matsuo <tmatsuo@google.com>
Source-Date: Fri Oct 16 09:58:05 2020 -0700
Source-Repo: googleapis/synthtool
Source-Sha: da5c6050d13b4950c82666a81d8acd25157664ae
Source-Link: googleapis/synthtool@da5c605

* chore(docs): update code of conduct of synthtool and templates

Source-Author: Christopher Wilcox <crwilcox@google.com>
Source-Date: Thu Oct 22 14:22:01 2020 -0700
Source-Repo: googleapis/synthtool
Source-Sha: 5f6ef0ec5501d33c4667885b37a7685a30d41a76
Source-Link: googleapis/synthtool@5f6ef0e

* docs: add proto-plus to intersphinx mapping

Source-Author: Tim Swast <swast@google.com>
Source-Date: Tue Oct 27 12:01:14 2020 -0500
Source-Repo: googleapis/synthtool
Source-Sha: ea52b8a0bd560f72f376efcf45197fb7c8869120
Source-Link: googleapis/synthtool@ea52b8a

* fix(python_library): fix external unit test dependencies

I recently submitted https://github.com/googleapis/synthtool/pull/811/files, allowing external dependencies for unit tests. This fixes a small missing comma bug

Source-Author: Daniel Sanche <d.sanche14@gmail.com>
Source-Date: Thu Oct 29 16:58:01 2020 -0700
Source-Repo: googleapis/synthtool
Source-Sha: 6542bd723403513626f61642fc02ddca528409aa
Source-Link: googleapis/synthtool@6542bd7

* chore: add type hint check

Source-Author: Leah E. Cole <6719667+leahecole@users.noreply.github.com>
Source-Date: Wed Nov 4 17:36:32 2020 -0800
Source-Repo: googleapis/synthtool
Source-Sha: 3d3e94c4e02370f307a9a200b0c743c3d8d19f29
Source-Link: googleapis/synthtool@3d3e94c

* chore: add blacken to template

Source-Author: Leah E. Cole <6719667+leahecole@users.noreply.github.com>
Source-Date: Thu Nov 5 15:22:03 2020 -0800
Source-Repo: googleapis/synthtool
Source-Sha: 1f1148d3c7a7a52f0c98077f976bd9b3c948ee2b
Source-Link: googleapis/synthtool@1f1148d

* fix: address lint issues

Source-Author: Leah E. Cole <6719667+leahecole@users.noreply.github.com>
Source-Date: Thu Nov 12 11:30:49 2020 -0800
Source-Repo: googleapis/synthtool
Source-Sha: e89175cf074dccc4babb4eca66ae913696e47a71
Source-Link: googleapis/synthtool@e89175c

packages/google-cloud-dataproc/.kokoro/docs/common.cfg

@@ -30,7 +30,7 @@ env_vars: {
 
 env_vars: {
     key: "V2_STAGING_BUCKET"
-    value: "docs-staging-v2-staging"
+    value: "docs-staging-v2"
 }
 
 # It will upload the docker image after successful builds.

packages/google-cloud-dataproc/.kokoro/populate-secrets.sh

@@ -0,0 +1,43 @@
+#!/bin/bash
+# Copyright 2020 Google LLC.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -eo pipefail
+
+function now { date +"%Y-%m-%d %H:%M:%S" | tr -d '\n' ;}
+function msg { println "$*" >&2 ;}
+function println { printf '%s\n' "$(now) $*" ;}
+
+
+# Populates requested secrets set in SECRET_MANAGER_KEYS from service account:
+# kokoro-trampoline@cloud-devrel-kokoro-resources.iam.gserviceaccount.com
+SECRET_LOCATION="${KOKORO_GFILE_DIR}/secret_manager"
+msg "Creating folder on disk for secrets: ${SECRET_LOCATION}"
+mkdir -p ${SECRET_LOCATION}
+for key in $(echo ${SECRET_MANAGER_KEYS} | sed "s/,/ /g")
+do
+  msg "Retrieving secret ${key}"
+  docker run --entrypoint=gcloud \
+    --volume=${KOKORO_GFILE_DIR}:${KOKORO_GFILE_DIR} \
+    gcr.io/google.com/cloudsdktool/cloud-sdk \
+    secrets versions access latest \
+    --project cloud-devrel-kokoro-resources \
+    --secret ${key} > \
+    "${SECRET_LOCATION}/${key}"
+  if [[ $? == 0 ]]; then
+    msg "Secret written to ${SECRET_LOCATION}/${key}"
+  else
+    msg "Error retrieving secret ${key}"
+  fi
+done

packages/google-cloud-dataproc/.kokoro/release/common.cfg

@@ -23,42 +23,18 @@ env_vars: {
     value: "github/python-dataproc/.kokoro/release.sh"
 }
 
-# Fetch the token needed for reporting release status to GitHub
-before_action {
-  fetch_keystore {
-    keystore_resource {
-      keystore_config_id: 73713
-      keyname: "yoshi-automation-github-key"
-    }
-  }
-}
-
-# Fetch PyPI password
-before_action {
-  fetch_keystore {
-    keystore_resource {
-      keystore_config_id: 73713
-      keyname: "google_cloud_pypi_password"
-    }
-  }
-}
-
-# Fetch magictoken to use with Magic Github Proxy 
-before_action {
-  fetch_keystore {
-    keystore_resource {
-      keystore_config_id: 73713
-      keyname: "releasetool-magictoken"
-    }
-  }
+# Fetch PyPI password	
+before_action {	
+  fetch_keystore {	
+    keystore_resource {	
+      keystore_config_id: 73713	
+      keyname: "google_cloud_pypi_password"	
+    }	
+  }	
 }
 
-# Fetch api key to use with Magic Github Proxy 
-before_action {
-  fetch_keystore {
-    keystore_resource {
-      keystore_config_id: 73713
-      keyname: "magic-github-proxy-api-key"
-    }
-  }
-}
+# Tokens needed to report release status back to GitHub
+env_vars: {
+  key: "SECRET_MANAGER_KEYS"
+  value: "releasetool-publish-reporter-app,releasetool-publish-reporter-googleapis-installation,releasetool-publish-reporter-pem"
+}

packages/google-cloud-dataproc/.kokoro/samples/python3.6/common.cfg

@@ -13,6 +13,12 @@ env_vars: {
     value: "py-3.6"
 }
 
+# Declare build specific Cloud project.
+env_vars: {
+    key: "BUILD_SPECIFIC_GCLOUD_PROJECT"
+    value: "python-docs-samples-tests-py36"
+}
+
 env_vars: {
     key: "TRAMPOLINE_BUILD_FILE"
     value: "github/python-dataproc/.kokoro/test-samples.sh"

packages/google-cloud-dataproc/.kokoro/samples/python3.7/common.cfg

@@ -13,6 +13,12 @@ env_vars: {
     value: "py-3.7"
 }
 
+# Declare build specific Cloud project.
+env_vars: {
+    key: "BUILD_SPECIFIC_GCLOUD_PROJECT"
+    value: "python-docs-samples-tests-py37"
+}
+
 env_vars: {
     key: "TRAMPOLINE_BUILD_FILE"
     value: "github/python-dataproc/.kokoro/test-samples.sh"

packages/google-cloud-dataproc/.kokoro/samples/python3.8/common.cfg

@@ -13,6 +13,12 @@ env_vars: {
     value: "py-3.8"
 }
 
+# Declare build specific Cloud project.
+env_vars: {
+    key: "BUILD_SPECIFIC_GCLOUD_PROJECT"
+    value: "python-docs-samples-tests-py38"
+}
+
 env_vars: {
     key: "TRAMPOLINE_BUILD_FILE"
     value: "github/python-dataproc/.kokoro/test-samples.sh"

packages/google-cloud-dataproc/.kokoro/test-samples.sh

@@ -28,6 +28,12 @@ if [[ $KOKORO_BUILD_ARTIFACTS_SUBDIR = *"periodic"* ]]; then
     git checkout $LATEST_RELEASE
 fi
 
+# Exit early if samples directory doesn't exist
+if [ ! -d "./samples" ]; then
+  echo "No tests run. `./samples` not found"
+  exit 0
+fi
+
 # Disable buffering, so that the logs stream through.
 export PYTHONUNBUFFERED=1
 
@@ -101,4 +107,4 @@ cd "$ROOT"
 # Workaround for Kokoro permissions issue: delete secrets
 rm testing/{test-env.sh,client-secrets.json,service-account.json}
 
-exit "$RTN"
+exit "$RTN"

packages/google-cloud-dataproc/.kokoro/trampoline.sh

@@ -15,9 +15,14 @@
 
 set -eo pipefail
 
-python3 "${KOKORO_GFILE_DIR}/trampoline_v1.py"  || ret_code=$?
+# Always run the cleanup script, regardless of the success of bouncing into
+# the container.
+function cleanup() {
+    chmod +x ${KOKORO_GFILE_DIR}/trampoline_cleanup.sh
+    ${KOKORO_GFILE_DIR}/trampoline_cleanup.sh
+    echo "cleanup";
+}
+trap cleanup EXIT
 
-chmod +x ${KOKORO_GFILE_DIR}/trampoline_cleanup.sh
-${KOKORO_GFILE_DIR}/trampoline_cleanup.sh || true
-
-exit ${ret_code}
+$(dirname $0)/populate-secrets.sh # Secret Manager secrets.
+python3 "${KOKORO_GFILE_DIR}/trampoline_v1.py"

packages/google-cloud-dataproc/CODE_OF_CONDUCT.md

@@ -1,44 +1,95 @@
 <!-- # Generated by synthtool. DO NOT EDIT! !-->
-# Contributor Code of Conduct
+# Code of Conduct
 
-As contributors and maintainers of this project,
-and in the interest of fostering an open and welcoming community,
-we pledge to respect all people who contribute through reporting issues,
-posting feature requests, updating documentation,
-submitting pull requests or patches, and other activities.
+## Our Pledge
 
-We are committed to making participation in this project
-a harassment-free experience for everyone,
-regardless of level of experience, gender, gender identity and expression,
-sexual orientation, disability, personal appearance,
-body size, race, ethnicity, age, religion, or nationality.
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to making participation in our project and
+our community a harassment-free experience for everyone, regardless of age, body
+size, disability, ethnicity, gender identity and expression, level of
+experience, education, socio-economic status, nationality, personal appearance,
+race, religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+*   Using welcoming and inclusive language
+*   Being respectful of differing viewpoints and experiences
+*   Gracefully accepting constructive criticism
+*   Focusing on what is best for the community
+*   Showing empathy towards other community members
 
 Examples of unacceptable behavior by participants include:
 
-* The use of sexualized language or imagery
-* Personal attacks
-* Trolling or insulting/derogatory comments
-* Public or private harassment
-* Publishing other's private information,
-such as physical or electronic
-addresses, without explicit permission
-* Other unethical or unprofessional conduct.
+*   The use of sexualized language or imagery and unwelcome sexual attention or
+    advances
+*   Trolling, insulting/derogatory comments, and personal or political attacks
+*   Public or private harassment
+*   Publishing others' private information, such as a physical or electronic
+    address, without explicit permission
+*   Other conduct which could reasonably be considered inappropriate in a
+    professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
 
 Project maintainers have the right and responsibility to remove, edit, or reject
-comments, commits, code, wiki edits, issues, and other contributions
-that are not aligned to this Code of Conduct.
-By adopting this Code of Conduct,
-project maintainers commit themselves to fairly and consistently
-applying these principles to every aspect of managing this project.
-Project maintainers who do not follow or enforce the Code of Conduct
-may be permanently removed from the project team.
-
-This code of conduct applies both within project spaces and in public spaces
-when an individual is representing the project or its community.
-
-Instances of abusive, harassing, or otherwise unacceptable behavior
-may be reported by opening an issue
-or contacting one or more of the project maintainers.
-
-This Code of Conduct is adapted from the [Contributor Covenant](http://contributor-covenant.org), version 1.2.0,
-available at [http://contributor-covenant.org/version/1/2/0/](http://contributor-covenant.org/version/1/2/0/)
+comments, commits, code, wiki edits, issues, and other contributions that are
+not aligned to this Code of Conduct, or to ban temporarily or permanently any
+contributor for other behaviors that they deem inappropriate, threatening,
+offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community. Examples of
+representing a project or community include using an official project e-mail
+address, posting via an official social media account, or acting as an appointed
+representative at an online or offline event. Representation of a project may be
+further defined and clarified by project maintainers.
+
+This Code of Conduct also applies outside the project spaces when the Project
+Steward has a reasonable belief that an individual's behavior may have a
+negative impact on the project or its community.
+
+## Conflict Resolution
+
+We do not believe that all conflict is bad; healthy debate and disagreement
+often yield positive results. However, it is never okay to be disrespectful or
+to engage in behavior that violates the project’s code of conduct.
+
+If you see someone violating the code of conduct, you are encouraged to address
+the behavior directly with those involved. Many issues can be resolved quickly
+and easily, and this gives people more control over the outcome of their
+dispute. If you are unable to resolve the matter for any reason, or if the
+behavior is threatening or harassing, report it. We are dedicated to providing
+an environment where participants feel welcome and safe.
+
+
+Reports should be directed to *googleapis-stewards@google.com*, the
+Project Steward(s) for *Google Cloud Client Libraries*. It is the Project Steward’s duty to
+receive and address reported violations of the code of conduct. They will then
+work with a committee consisting of representatives from the Open Source
+Programs Office and the Google Open Source Strategy team. If for any reason you
+are uncomfortable reaching out to the Project Steward, please email
+opensource@google.com.
+
+We will investigate every complaint, but you may not receive a direct response.
+We will use our discretion in determining when and how to follow up on reported
+incidents, which may range from not taking action to permanent expulsion from
+the project and project-sponsored spaces. We will notify the accused of the
+report and provide them an opportunity to discuss it before any action is taken.
+The identity of the reporter will be omitted from the details of the report
+supplied to the accused. In potentially harmful situations, such as ongoing
+harassment or threats to anyone's safety, we may take action without notice.
+
+## Attribution
+
+This Code of Conduct is adapted from the Contributor Covenant, version 1.4,
+available at
+https://www.contributor-covenant.org/version/1/4/code-of-conduct.html

packages/google-cloud-dataproc/CONTRIBUTING.rst

@@ -80,25 +80,6 @@ We use `nox <https://nox.readthedocs.io/en/latest/>`__ to instrument our tests.
 
 .. nox: https://pypi.org/project/nox/
 
-Note on Editable Installs / Develop Mode
-========================================
-
-- As mentioned previously, using ``setuptools`` in `develop mode`_
-  or a ``pip`` `editable install`_ is not possible with this
-  library. This is because this library uses `namespace packages`_.
-  For context see `Issue #2316`_ and the relevant `PyPA issue`_.
-
-  Since ``editable`` / ``develop`` mode can't be used, packages
-  need to be installed directly. Hence your changes to the source
-  tree don't get incorporated into the **already installed**
-  package.
-
-.. _namespace packages: https://www.python.org/dev/peps/pep-0420/
-.. _Issue #2316: https://github.com/GoogleCloudPlatform/google-cloud-python/issues/2316
-.. _PyPA issue: https://github.com/pypa/packaging-problems/issues/12
-.. _develop mode: https://setuptools.readthedocs.io/en/latest/setuptools.html#development-mode
-.. _editable install: https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs
-
 *****************************************
 I'm getting weird errors... Can you help?
 *****************************************

packages/google-cloud-dataproc/docs/conf.py

@@ -29,7 +29,7 @@
 # -- General configuration ------------------------------------------------
 
 # If your documentation needs a minimal Sphinx version, state it here.
-needs_sphinx = "1.6.3"
+needs_sphinx = "1.5.5"
 
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
@@ -39,6 +39,7 @@
     "sphinx.ext.autosummary",
     "sphinx.ext.intersphinx",
     "sphinx.ext.coverage",
+    "sphinx.ext.doctest",
     "sphinx.ext.napoleon",
     "sphinx.ext.todo",
     "sphinx.ext.viewcode",
@@ -348,6 +349,7 @@
     "google-auth": ("https://google-auth.readthedocs.io/en/stable", None),
     "google.api_core": ("https://googleapis.dev/python/google-api-core/latest/", None,),
     "grpc": ("https://grpc.io/grpc/python/", None),
+    "proto-plus": ("https://proto-plus-python.readthedocs.io/en/latest/", None),
 }
 
 

packages/google-cloud-dataproc/docs/dataproc_v1/types.rst

@@ -3,3 +3,4 @@ Types for Google Cloud Dataproc v1 API
 
 .. automodule:: google.cloud.dataproc_v1.types
     :members:
+    :show-inheritance:

packages/google-cloud-dataproc/docs/dataproc_v1beta2/types.rst

@@ -3,3 +3,4 @@ Types for Google Cloud Dataproc v1beta2 API
 
 .. automodule:: google.cloud.dataproc_v1beta2.types
     :members:
+    :show-inheritance: