feat: [google-cloud-dataproc] add resource reference for KMS keys and…

… fix comments (#13072) - [ ] Regenerate this pull request now. BEGIN_COMMIT_OVERRIDE feat: [google-cloud-dataproc] add resource reference for KMS keys and fix comments feat: [google-cloud-dataproc] Allow flink and trino job support for workflow templates API feat: [google-cloud-dataproc] Add unreachable output field for LIST workflow template API feat: [google-cloud-dataproc] Add unreachable output field for LIST batch templates API feat: [google-cloud-dataproc] Add kms key input for create cluster API feat: [google-cloud-dataproc] Add FLINK metric source for Dataproc Metric Source feat: [google-cloud-dataproc] Allow flink job support for jobs feat: [google-cloud-dataproc] Add unreachable output field for LIST jobs API END_COMMIT_OVERRIDE PiperOrigin-RevId: 674408200 Source-Link: googleapis/googleapis@964f6c9 Source-Link: googleapis/googleapis-gen@2979ec5 Copy-Tag: eyJwIjoicGFja2FnZXMvZ29vZ2xlLWNsb3VkLWRhdGFwcm9jLy5Pd2xCb3QueWFtbCIsImgiOiIyOTc5ZWM1Y2JjZjJhYmFlODU3ZjlhNGNiZTVjY2EwOWYwNDBjYjU4In0= BEGIN_NESTED_COMMIT feat: [google-cloud-dataproc] add support for new Dataproc features 1. Allow flink and trino job support for workflow templates API 2. Add unreachable output field for LIST workflow template API 4. Add unreachable output field for LIST batch templates API 5. Add kms key input for create cluster API 6. Add FLINK metric source for Dataproc Metric Source PiperOrigin-RevId: 673000575 Source-Link: googleapis/googleapis@02f62c8 Source-Link: googleapis/googleapis-gen@7726f47 Copy-Tag: eyJwIjoicGFja2FnZXMvZ29vZ2xlLWNsb3VkLWRhdGFwcm9jLy5Pd2xCb3QueWFtbCIsImgiOiI3NzI2ZjQ3OGFjMTlkOTFlOTE0ZWQzYWU1NDZjZjI0NDUzZDAwMGI1In0= END_NESTED_COMMIT BEGIN_NESTED_COMMIT feat: [google-cloud-dataproc] add support for new Dataproc features 1. Allow flink job support for jobs 2. Add unreachable output field for LIST jobs API PiperOrigin-RevId: 672705294 Source-Link: googleapis/googleapis@32bc036 Source-Link: googleapis/googleapis-gen@46e7728 Copy-Tag: eyJwIjoicGFja2FnZXMvZ29vZ2xlLWNsb3VkLWRhdGFwcm9jLy5Pd2xCb3QueWFtbCIsImgiOiI0NmU3NzI4Yzk5MDhkOTc5M2ViY2UxMDYxYjBkMWM2YzRiYWQ5MjViIn0= END_NESTED_COMMIT --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Victor Chudnovsky <vchudnov@google.com>
googleapis · Sep 14, 2024 · 2402404 · 2402404
1 parent 77b64f5
commit 2402404
Show file tree

Hide file tree

Showing 19 changed files with 598 additions and 138 deletions.
diff --git a/packages/google-cloud-dataproc/google/cloud/dataproc/__init__.py b/packages/google-cloud-dataproc/google/cloud/dataproc/__init__.py
@@ -135,6 +135,7 @@
     CancelJobRequest,
     DeleteJobRequest,
     DriverSchedulingConfig,
+    FlinkJob,
     GetJobRequest,
     HadoopJob,
     HiveJob,
@@ -315,6 +316,7 @@
     "CancelJobRequest",
     "DeleteJobRequest",
     "DriverSchedulingConfig",
+    "FlinkJob",
     "GetJobRequest",
     "HadoopJob",
     "HiveJob",

diff --git a/packages/google-cloud-dataproc/google/cloud/dataproc/gapic_version.py b/packages/google-cloud-dataproc/google/cloud/dataproc/gapic_version.py
@@ -13,4 +13,4 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
-__version__ = "5.11.0"  # {x-release-please-version}
+__version__ = "0.0.0"  # {x-release-please-version}
diff --git a/packages/google-cloud-dataproc/google/cloud/dataproc_v1/__init__.py b/packages/google-cloud-dataproc/google/cloud/dataproc_v1/__init__.py
@@ -115,6 +115,7 @@
     CancelJobRequest,
     DeleteJobRequest,
     DriverSchedulingConfig,
+    FlinkJob,
     GetJobRequest,
     HadoopJob,
     HiveJob,
@@ -271,6 +272,7 @@
     "EnvironmentConfig",
     "ExecutionConfig",
     "FailureAction",
+    "FlinkJob",
     "GceClusterConfig",
     "GetAutoscalingPolicyRequest",
     "GetBatchRequest",

diff --git a/packages/google-cloud-dataproc/google/cloud/dataproc_v1/gapic_version.py b/packages/google-cloud-dataproc/google/cloud/dataproc_v1/gapic_version.py
@@ -13,4 +13,4 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
-__version__ = "5.11.0"  # {x-release-please-version}
+__version__ = "0.0.0"  # {x-release-please-version}
diff --git a/...oogle-cloud-dataproc/google/cloud/dataproc_v1/services/cluster_controller/async_client.py b/...oogle-cloud-dataproc/google/cloud/dataproc_v1/services/cluster_controller/async_client.py
@@ -74,6 +74,8 @@ class ClusterControllerAsyncClient:
 
     cluster_path = staticmethod(ClusterControllerClient.cluster_path)
     parse_cluster_path = staticmethod(ClusterControllerClient.parse_cluster_path)
+    crypto_key_path = staticmethod(ClusterControllerClient.crypto_key_path)
+    parse_crypto_key_path = staticmethod(ClusterControllerClient.parse_crypto_key_path)
     node_group_path = staticmethod(ClusterControllerClient.node_group_path)
     parse_node_group_path = staticmethod(ClusterControllerClient.parse_node_group_path)
     service_path = staticmethod(ClusterControllerClient.service_path)
@@ -1236,10 +1238,11 @@ async def sample_list_clusters():
                 label key. **value** can be ``*`` to match all values.
                 ``status.state`` can be one of the following:
                 ``ACTIVE``, ``INACTIVE``, ``CREATING``, ``RUNNING``,
-                ``ERROR``, ``DELETING``, or ``UPDATING``. ``ACTIVE``
-                contains the ``CREATING``, ``UPDATING``, and ``RUNNING``
-                states. ``INACTIVE`` contains the ``DELETING`` and
-                ``ERROR`` states. ``clusterName`` is the name of the
+                ``ERROR``, ``DELETING``, ``UPDATING``, ``STOPPING``, or
+                ``STOPPED``. ``ACTIVE`` contains the ``CREATING``,
+                ``UPDATING``, and ``RUNNING`` states. ``INACTIVE``
+                contains the ``DELETING``, ``ERROR``, ``STOPPING``, and
+                ``STOPPED`` states. ``clusterName`` is the name of the
                 cluster provided at creation time. Only the logical
                 ``AND`` operator is supported; space-separated items are
                 treated as having an implicit ``AND`` operator.

diff --git a/...ages/google-cloud-dataproc/google/cloud/dataproc_v1/services/cluster_controller/client.py b/...ages/google-cloud-dataproc/google/cloud/dataproc_v1/services/cluster_controller/client.py
@@ -215,6 +215,30 @@ def parse_cluster_path(path: str) -> Dict[str, str]:
         )
         return m.groupdict() if m else {}
 
+    @staticmethod
+    def crypto_key_path(
+        project: str,
+        location: str,
+        key_ring: str,
+        crypto_key: str,
+    ) -> str:
+        """Returns a fully-qualified crypto_key string."""
+        return "projects/{project}/locations/{location}/keyRings/{key_ring}/cryptoKeys/{crypto_key}".format(
+            project=project,
+            location=location,
+            key_ring=key_ring,
+            crypto_key=crypto_key,
+        )
+
+    @staticmethod
+    def parse_crypto_key_path(path: str) -> Dict[str, str]:
+        """Parses a crypto_key path into its component segments."""
+        m = re.match(
+            r"^projects/(?P<project>.+?)/locations/(?P<location>.+?)/keyRings/(?P<key_ring>.+?)/cryptoKeys/(?P<crypto_key>.+?)$",
+            path,
+        )
+        return m.groupdict() if m else {}
+
     @staticmethod
     def node_group_path(
         project: str,
@@ -1686,10 +1710,11 @@ def sample_list_clusters():
                 label key. **value** can be ``*`` to match all values.
                 ``status.state`` can be one of the following:
                 ``ACTIVE``, ``INACTIVE``, ``CREATING``, ``RUNNING``,
-                ``ERROR``, ``DELETING``, or ``UPDATING``. ``ACTIVE``
-                contains the ``CREATING``, ``UPDATING``, and ``RUNNING``
-                states. ``INACTIVE`` contains the ``DELETING`` and
-                ``ERROR`` states. ``clusterName`` is the name of the
+                ``ERROR``, ``DELETING``, ``UPDATING``, ``STOPPING``, or
+                ``STOPPED``. ``ACTIVE`` contains the ``CREATING``,
+                ``UPDATING``, and ``RUNNING`` states. ``INACTIVE``
+                contains the ``DELETING``, ``ERROR``, ``STOPPING``, and
+                ``STOPPED`` states. ``clusterName`` is the name of the
                 cluster provided at creation time. Only the logical
                 ``AND`` operator is supported; space-separated items are
                 treated as having an implicit ``AND`` operator.

diff --git a/...loud-dataproc/google/cloud/dataproc_v1/services/workflow_template_service/async_client.py b/...loud-dataproc/google/cloud/dataproc_v1/services/workflow_template_service/async_client.py
@@ -75,6 +75,10 @@ class WorkflowTemplateServiceAsyncClient:
     )
     _DEFAULT_UNIVERSE = WorkflowTemplateServiceClient._DEFAULT_UNIVERSE
 
+    crypto_key_path = staticmethod(WorkflowTemplateServiceClient.crypto_key_path)
+    parse_crypto_key_path = staticmethod(
+        WorkflowTemplateServiceClient.parse_crypto_key_path
+    )
     node_group_path = staticmethod(WorkflowTemplateServiceClient.node_group_path)
     parse_node_group_path = staticmethod(
         WorkflowTemplateServiceClient.parse_node_group_path

diff --git a/...ogle-cloud-dataproc/google/cloud/dataproc_v1/services/workflow_template_service/client.py b/...ogle-cloud-dataproc/google/cloud/dataproc_v1/services/workflow_template_service/client.py
@@ -194,6 +194,30 @@ def transport(self) -> WorkflowTemplateServiceTransport:
         """
         return self._transport
 
+    @staticmethod
+    def crypto_key_path(
+        project: str,
+        location: str,
+        key_ring: str,
+        crypto_key: str,
+    ) -> str:
+        """Returns a fully-qualified crypto_key string."""
+        return "projects/{project}/locations/{location}/keyRings/{key_ring}/cryptoKeys/{crypto_key}".format(
+            project=project,
+            location=location,
+            key_ring=key_ring,
+            crypto_key=crypto_key,
+        )
+
+    @staticmethod
+    def parse_crypto_key_path(path: str) -> Dict[str, str]:
+        """Parses a crypto_key path into its component segments."""
+        m = re.match(
+            r"^projects/(?P<project>.+?)/locations/(?P<location>.+?)/keyRings/(?P<key_ring>.+?)/cryptoKeys/(?P<crypto_key>.+?)$",
+            path,
+        )
+        return m.groupdict() if m else {}
+
     @staticmethod
     def node_group_path(
         project: str,

diff --git a/packages/google-cloud-dataproc/google/cloud/dataproc_v1/types/__init__.py b/packages/google-cloud-dataproc/google/cloud/dataproc_v1/types/__init__.py
@@ -84,6 +84,7 @@
     CancelJobRequest,
     DeleteJobRequest,
     DriverSchedulingConfig,
+    FlinkJob,
     GetJobRequest,
     HadoopJob,
     HiveJob,
@@ -248,6 +249,7 @@
     "CancelJobRequest",
     "DeleteJobRequest",
     "DriverSchedulingConfig",
+    "FlinkJob",
     "GetJobRequest",
     "HadoopJob",
     "HiveJob",

diff --git a/packages/google-cloud-dataproc/google/cloud/dataproc_v1/types/batches.py b/packages/google-cloud-dataproc/google/cloud/dataproc_v1/types/batches.py
@@ -183,6 +183,11 @@ class ListBatchesResponse(proto.Message):
             A token, which can be sent as ``page_token`` to retrieve the
             next page. If this field is omitted, there are no subsequent
             pages.
+        unreachable (MutableSequence[str]):
+            Output only. List of Batches that could not
+            be included in the response. Attempting to get
+            one of these resources may indicate why it was
+            not included in the list response.
     """
 
     @property
@@ -198,6 +203,10 @@ def raw_page(self):
         proto.STRING,
         number=2,
     )
+    unreachable: MutableSequence[str] = proto.RepeatedField(
+        proto.STRING,
+        number=3,
+    )
 
 
 class DeleteBatchRequest(proto.Message):

diff --git a/packages/google-cloud-dataproc/google/cloud/dataproc_v1/types/clusters.py b/packages/google-cloud-dataproc/google/cloud/dataproc_v1/types/clusters.py
@@ -464,15 +464,50 @@ class EncryptionConfig(proto.Message):
 
     Attributes:
         gce_pd_kms_key_name (str):
-            Optional. The Cloud KMS key name to use for
-            PD disk encryption for all instances in the
-            cluster.
+            Optional. The Cloud KMS key resource name to use for
+            persistent disk encryption for all instances in the cluster.
+            See [Use CMEK with cluster data]
+            (https://cloud.google.com//dataproc/docs/concepts/configuring-clusters/customer-managed-encryption#use_cmek_with_cluster_data)
+            for more information.
+        kms_key (str):
+            Optional. The Cloud KMS key resource name to use for cluster
+            persistent disk and job argument encryption. See [Use CMEK
+            with cluster data]
+            (https://cloud.google.com//dataproc/docs/concepts/configuring-clusters/customer-managed-encryption#use_cmek_with_cluster_data)
+            for more information.
+
+            When this key resource name is provided, the following job
+            arguments of the following job types submitted to the
+            cluster are encrypted using CMEK:
+
+            -  `FlinkJob
+               args <https://cloud.google.com/dataproc/docs/reference/rest/v1/FlinkJob>`__
+            -  `HadoopJob
+               args <https://cloud.google.com/dataproc/docs/reference/rest/v1/HadoopJob>`__
+            -  `SparkJob
+               args <https://cloud.google.com/dataproc/docs/reference/rest/v1/SparkJob>`__
+            -  `SparkRJob
+               args <https://cloud.google.com/dataproc/docs/reference/rest/v1/SparkRJob>`__
+            -  `PySparkJob
+               args <https://cloud.google.com/dataproc/docs/reference/rest/v1/PySparkJob>`__
+            -  `SparkSqlJob <https://cloud.google.com/dataproc/docs/reference/rest/v1/SparkSqlJob>`__
+               scriptVariables and queryList.queries
+            -  `HiveJob <https://cloud.google.com/dataproc/docs/reference/rest/v1/HiveJob>`__
+               scriptVariables and queryList.queries
+            -  `PigJob <https://cloud.google.com/dataproc/docs/reference/rest/v1/PigJob>`__
+               scriptVariables and queryList.queries
+            -  `PrestoJob <https://cloud.google.com/dataproc/docs/reference/rest/v1/PrestoJob>`__
+               scriptVariables and queryList.queries
     """
 
     gce_pd_kms_key_name: str = proto.Field(
         proto.STRING,
         number=1,
     )
+    kms_key: str = proto.Field(
+        proto.STRING,
+        number=2,
+    )
 
 
 class GceClusterConfig(proto.Message):
@@ -519,14 +554,25 @@ class GceClusterConfig(proto.Message):
             -  ``projects/[project_id]/regions/[region]/subnetworks/sub0``
             -  ``sub0``
         internal_ip_only (bool):
-            Optional. If true, all instances in the cluster will only
-            have internal IP addresses. By default, clusters are not
-            restricted to internal IP addresses, and will have ephemeral
-            external IP addresses assigned to each instance. This
-            ``internal_ip_only`` restriction can only be enabled for
-            subnetwork enabled networks, and all off-cluster
-            dependencies must be configured to be accessible without
-            external IP addresses.
+            Optional. This setting applies to subnetwork-enabled
+            networks. It is set to ``true`` by default in clusters
+            created with image versions 2.2.x.
+
+            When set to ``true``:
+
+            -  All cluster VMs have internal IP addresses.
+            -  [Google Private Access]
+               (https://cloud.google.com/vpc/docs/private-google-access)
+               must be enabled to access Dataproc and other Google Cloud
+               APIs.
+            -  Off-cluster dependencies must be configured to be
+               accessible without external IP addresses.
+
+            When set to ``false``:
+
+            -  Cluster VMs are not restricted to internal IP addresses.
+            -  Ephemeral external IP addresses are assigned to each
+               cluster VM.
 
             This field is a member of `oneof`_ ``_internal_ip_only``.
         private_ipv6_google_access (google.cloud.dataproc_v1.types.GceClusterConfig.PrivateIpv6GoogleAccess):
@@ -560,9 +606,9 @@ class GceClusterConfig(proto.Message):
             -  https://www.googleapis.com/auth/bigtable.data
             -  https://www.googleapis.com/auth/devstorage.full_control
         tags (MutableSequence[str]):
-            The Compute Engine tags to add to all instances (see
+            The Compute Engine network tags to add to all instances (see
             `Tagging
-            instances <https://cloud.google.com/compute/docs/label-or-tag-resources#tags>`__).
+            instances <https://cloud.google.com/vpc/docs/add-remove-network-tags>`__).
         metadata (MutableMapping[str, str]):
             Optional. The Compute Engine metadata entries to add to all
             instances (see `Project and instance
@@ -1156,15 +1202,15 @@ class AcceleratorConfig(proto.Message):
 
             Examples:
 
-            -  ``https://www.googleapis.com/compute/v1/projects/[project_id]/zones/[zone]/acceleratorTypes/nvidia-tesla-k80``
-            -  ``projects/[project_id]/zones/[zone]/acceleratorTypes/nvidia-tesla-k80``
-            -  ``nvidia-tesla-k80``
+            -  ``https://www.googleapis.com/compute/v1/projects/[project_id]/zones/[zone]/acceleratorTypes/nvidia-tesla-t4``
+            -  ``projects/[project_id]/zones/[zone]/acceleratorTypes/nvidia-tesla-t4``
+            -  ``nvidia-tesla-t4``
 
             **Auto Zone Exception**: If you are using the Dataproc `Auto
             Zone
             Placement <https://cloud.google.com/dataproc/docs/concepts/configuring-clusters/auto-zone#using_auto_zone_placement>`__
             feature, you must use the short name of the accelerator type
-            resource, for example, ``nvidia-tesla-k80``.
+            resource, for example, ``nvidia-tesla-t4``.
         accelerator_count (int):
             The number of the accelerator cards of this
             type exposed to this instance.
@@ -1501,8 +1547,8 @@ class KerberosConfig(proto.Message):
             encrypted file containing the root principal
             password.
         kms_key_uri (str):
-            Optional. The uri of the KMS key used to
-            encrypt various sensitive files.
+            Optional. The URI of the KMS key used to
+            encrypt sensitive files.
         keystore_uri (str):
             Optional. The Cloud Storage URI of the
             keystore file used for SSL encryption. If not
@@ -1649,7 +1695,7 @@ class SoftwareConfig(proto.Message):
         image_version (str):
             Optional. The version of software inside the cluster. It
             must be one of the supported `Dataproc
-            Versions <https://cloud.google.com/dataproc/docs/concepts/versioning/dataproc-versions#supported_dataproc_versions>`__,
+            Versions <https://cloud.google.com/dataproc/docs/concepts/versioning/dataproc-versions#supported-dataproc-image-versions>`__,
             such as "1.2" (including a subminor version, such as
             "1.2.29"), or the `"preview"
             version <https://cloud.google.com/dataproc/docs/concepts/versioning/dataproc-versions#other_versions>`__.
@@ -1834,6 +1880,8 @@ class MetricSource(proto.Enum):
                 Hiveserver2 metric source.
             HIVEMETASTORE (7):
                 hivemetastore metric source
+            FLINK (8):
+                flink metric source
         """
         METRIC_SOURCE_UNSPECIFIED = 0
         MONITORING_AGENT_DEFAULTS = 1
@@ -1843,6 +1891,7 @@ class MetricSource(proto.Enum):
         SPARK_HISTORY_SERVER = 5
         HIVESERVER2 = 6
         HIVEMETASTORE = 7
+        FLINK = 8
 
     class Metric(proto.Message):
         r"""A Dataproc custom metric.
@@ -2312,11 +2361,12 @@ class ListClustersRequest(proto.Message):
             or ``labels.[KEY]``, and ``[KEY]`` is a label key. **value**
             can be ``*`` to match all values. ``status.state`` can be
             one of the following: ``ACTIVE``, ``INACTIVE``,
-            ``CREATING``, ``RUNNING``, ``ERROR``, ``DELETING``, or
-            ``UPDATING``. ``ACTIVE`` contains the ``CREATING``,
-            ``UPDATING``, and ``RUNNING`` states. ``INACTIVE`` contains
-            the ``DELETING`` and ``ERROR`` states. ``clusterName`` is
-            the name of the cluster provided at creation time. Only the
+            ``CREATING``, ``RUNNING``, ``ERROR``, ``DELETING``,
+            ``UPDATING``, ``STOPPING``, or ``STOPPED``. ``ACTIVE``
+            contains the ``CREATING``, ``UPDATING``, and ``RUNNING``
+            states. ``INACTIVE`` contains the ``DELETING``, ``ERROR``,
+            ``STOPPING``, and ``STOPPED`` states. ``clusterName`` is the
+            name of the cluster provided at creation time. Only the
             logical ``AND`` operator is supported; space-separated items
             are treated as having an implicit ``AND`` operator.
 
@@ -2393,10 +2443,10 @@ class DiagnoseClusterRequest(proto.Message):
         cluster_name (str):
             Required. The cluster name.
         tarball_gcs_dir (str):
-            Optional. The output Cloud Storage directory
-            for the diagnostic tarball. If not specified, a
-            task-specific directory in the cluster's staging
-            bucket will be used.
+            Optional. (Optional) The output Cloud Storage
+            directory for the diagnostic tarball. If not
+            specified, a task-specific directory in the
+            cluster's staging bucket will be used.
         tarball_access (google.cloud.dataproc_v1.types.DiagnoseClusterRequest.TarballAccess):
             Optional. (Optional) The access type to the
             diagnostic tarball. If not specified, falls back