Merge from master. (#4468)

* Bump next snapshot (#4300)

* Cloud Bigtable: instanceAdmin sample (#4299)

* instanceAdmin sample and tests

* comments added

* instanceAdminExample changes

* renamed variables

* changes

* Regenerate securitycenter client (#4311)

* 3540: Used transformer for shaded plugin (#4305)

* Added Query#fromProto to convert protobuf ReadRowsRequest (#4291)

* Adding Query.fromProto

* Adding Query.fromProto

* adding changes as per feedback

* updated javadoc as per feedback

* reformatted Query & QueryTest.java

* Bigquery: corrected equality check on subclasses of StringEnumValue (#4283)

* Bigquery: corrected equality check on subclasses of StringEnumValue

* update format for FieldTest.java

* Fix #4269 update metata url to FQDN (#4278)

* 4152: Added checking PAGE_TOKEN from options (#4303)

* 3918: Added checking of attributes size for 0. Added unit test (#4304)

* Bigtable: Merge bigtable-admin into the bigtable artifact (#4307)

* Bigtable: merge bigtable-admin into the bigtable artifact

* split exclusion change for future PR

* fix admin pathes

* fix deprecation warning

* fix synth script

* fix generated file

* temporarily re-add kokoro scripts (and point them to the merged artifact) until the jobs have been removed

* Remove dep from examples

* fix admin integration tests

* revert stray fix (will be added to a separate PR)

* fix int tests

* don't double format the code

* fix up docs

* Regenerate PubSub: documentation updates (#4293)

* Regenerate monitoring client (#4316)

* Regenerate bigtable client (#4318)

* 3822: Fixed setDisableGZipContent call (#4314)

* 3822: Fixed setDisableGZipContent call

* 3822: Changes after review

* Regenerate speech client: enable multi-channel features (#4317)

* Release v0.77.0 (#4324)

* Created enum Region.java to retrieve Regions without doing request. (#4309)

* Bump next snapshot (#4325)

* Bigtable: fix handling of unset rpc timeouts (#4323)

When the rpc timeout is zero, then it should be treated as disabled not actual 0

* Bigtable: Expose single row read settings (#4321)

* exposing ReadRowSettings thru BigtableDataSettings

* fixed typo error

* Regenerate compute client (#4327)

* Regenerate speech client (#4328)

* Update README version to use bom version (#4322)

* BigQuery: Fix NPE for null table schema fields (#4338)

* Add failing test for empty table schema

* Fix NPE if table schema returns null for fields

* Bump gax, grpc & opencensus version (#4336)

* Cloud Bigtable: HelloWorld sample updates (#4339)

* comments added in HelloWorld and ITHelloWorld

* removed typsafe name

* separate properties for bigtable.project and bigtable.instance

* separate properties for bigtable.project and bigtable.instance (#4346)

* Pub/Sub: default values in batch settings comments (#4350)

* Pub/Sub: default values in batch settings comments

Ref: https://github.com/googleapis/gax-java/blob/master/gax/src/main/java/com/google/api/gax/batching/BatchingSettings.java

* Verbose comment

* Update maven-surefire-plugin to 3.0.0-M3 to fix Java 8 classloader (#4344)

* Update maven-surefire-plugin to 3.0.0-M3 to fix Java 8 classloader

* Update failsafe plugin to 3.0.0-M3 too

* Specify maven-surefire-plugin version in bigtable-emulator

* Bigtable: Fixing a typo for KeyOffSet#geyKey to getKey (#4342)

* Regenerate spanner client (#4332)

* Spanner: remove junit from compile dependencies (#4334)

* Firestore: change timestampsInSnapshots default to true. (#4353)

BREAKING CHANGE: The `areTimestampsInSnapshotsEnabled()` setting is now enabled
by default so timestamp fields read from a `DocumentSnapshot` will be returned
as `Timestamp` objects instead of `Date`. Any code expecting to receive a
`Date` object must be updated.

* Regenerate clients with updated copyright year (#4382)

* Regenerate asset client

* Regenerate automl client

* Regenerate bigquerydatatransfer client

* Regenerate bigquerystorage client

* Regenerate bigtable client

* Regenerate container client

* Regenerate containeranalysis client

* Regenerate dataproc client

* Regenerate dialogflow client

* Regenerate dlp client

* Regenerate errorreporting client

* Regenerate iamcredentials client

* Regenerate iot client

* Regenerate kms client

* Regenerate language client

* Regenerate logging client

* Regenerate monitoring client

* Regenerate os-login client

* Regenerate pubsub client

* Regenerate redis client

* Regenerate securitycenter client

* Regenerate speech client

* Regenerate tasks client

* Regenerate trace client

* Regenerate video-intelligence client

* Regenerate websecurityscanner client

* Release google-cloud-java v0.78.0 (#4386)

* Regenerate compute client (#4359)

* Cloud Bigtable: tableAdmin sample (#4313)

* tableAdmin sample and tests

* comments added

* files renamed and other edits

* some changes in TableAdminExample and tests

* fixed timestamp to base 16

* separate properties for bigtable.project and bigtable.instance

* Regenerate scheduler client (#4294)

* Regenerate spanner client (#4388)

* Bump next snapshot (#4391)

* Regenerate asset client (#4395)

* Regenerate scheduler client (#4396)

* Firestore: Include a trailing /documents on root resource paths (#4352)

This is required for v1 and accepted in v1beta1.

Port of googleapis/nodejs-firestore@52c7381

* Release v0.79.0 (#4402)

* Removing some unused dependencies (#4385)

* Removing some unused dependencies
Also, reducing scope of auto-value to provided.

* Restoring Firestore auto-value

* Removing more instances of easymock.

* Removing non-deprecated uses of joda time. (#4351)

* Removing non-deprecated uses of joda time.
This works towards #3482

* Update pom.xml

* Ran `mvn com.coveo:fmt-maven-plugin:format`

* Fixing a bad merge

* Bump next snapshot (#4405)

* fix shading (#4406)

* Fixing some deprecation warnings (#4390)

* fixing some deprecation warnings

* updated comment

* BigQuery : Fix Location configurable at BigQueryOptions (#4329)

* Fix Location configurable from BigQueryOptions

* modified code

* modified code and add test case

* removed unused location

* Generate Firestore API v1 (#4410)

* Bigtable: make row & cell ordering more consistent. (#4421)

* Bigtable: make row & cell ordering more consistent.

* RowCells should always be ordered in lexicographically by family, then qualifier and finally by reverse chronological order
* Although rows will always be ordered lexicographically by row key, they should not implement Comparable to avoid confusion when compareTo() == 0 and equals() is false. Instead that ordering was moved to a separate comparator.

* Add helpers to filter cells by family & qualifier

* tweaks

* code style

* code style

* Fix code formatting (#4437)

* Regenerate compute client (#4399)

* Regenerate compute client (#4444)

* Firestore: Migrate client to use v1 gapic client / protos. (#4419)

* Regenerate asset client (#4426)

* Regenerate bigtable client (#4427)

* Regenerate dataproc client (#4428)

* Regenerate speech client (#4422)

* Regenerate dialogflow client (#4429)

* Regenerate redis client (#4431)

* Regenerate securitycenter client (#4432)

* Regenerate tasks client (#4411)

* Fix usages of any(<Primitive>.class) matchers (#4453)

In Mockito 2, if a method expects a primitive type, but an any(<Primitive>.class) matcher is used in its place, it will throw an error. To prepare for this upcoming breakage, change
all existing any(<Primitive>.class) matchers to use the correct any<Primitive>() matcher.

* Regenerate video-intelligence client (#4434)

* Regenerate automl client (#4418)

* Regenerate automl client (#4455)

* Regenerate speech client (#4456)

* add comments (#4441)

* Update some default retry setting values (#4425)

Ref: https://github.com/googleapis/google-cloud-java/blob/ed7bc857a05b34eed6be182d4798a62bf09cd394/google-cloud-clients/google-cloud-pubsub/src/main/java/com/google/cloud/pubsub/v1/Publisher.java#L497

* Implement BucketPolicyOnly (#4404)

* implement BucketPolicyOnly

* [Storage] Bucket Policy Only samples (#4408)

* Humble beginnings for BPO samples with tests

* Update samples per client changes

* Fix format issue

* Lint fix

* Bigtable: make request context serializable (#4459)

* Bigtable: expose helper to convert proto rows into model rows (#4458)

This is needed for hbase adapter transition

* Regenerate texttospeech client (#4333)

* Regenerate compute client (#4462)

* Skip integration tests if no changes (#4392)

* Add environment variable for allowing skipping ITs

* Skip integration tests if there are no changes in that directory

* Also check google-cloud-core client and the grpc-api generated models

* update TESTING.md (#4409)

* Regenerate spanner client (#4433)

* Regenerate texttospeech client (#4463)

* 3534: Made HttpTransportOptions constructor public. (#4439)

* use gax 1.38.0 (#4460)

* Regenerate firestore client (#4348)

* Empty table results can have a schema (#4185)

* Update EmptyTableResult.java

* Update Job.java

* add test case

* 4273: Added a new create method to pass offset and length of sub array. (#4407)

* 4273: Added a new create method to pass offset and lenght of sub array.

* 4273: Fixed codeformat error.

* 4273: Rephrased a comment.

* 4273: Added a new integration test using the new createBlobWithSubArrayFromByteArray code snippet.

* Add firestore grpc and proto versions to versions.txt (#4464)

* Release v0.80.0 (#4465)

* Bump next snapshot (#4467)

* Adding Cloud Bigtable Mutation fromProto (#4461)

* Adding Cloud Bigtable Mutation fromProto

* Fixing formatting

* Addressing comments
`fromProto` becomes `fromProtoUnsafe`

* Fixing lint issues.

* adding a warning to `fromProtoUnsafe`

* Regenerate websecurityscanner client (#4435)

Author: kamalaboulhosn

.kokoro/build.sh

@@ -17,14 +17,44 @@ set -eo pipefail
 
 cd github/google-cloud-java/
 
+function client_has_changes() {
+  CLIENT_NAME=$1
+  if [[ ! -z $(git diff master google-cloud-clients/google-cloud-core*) ]]; then
+    echo "true"
+    return
+  fi
+
+  if [[ ! -z $(git diff master google-cloud-clients/${CLIENT_NAME}) ]]; then
+    echo "true"
+    return
+  fi
+
+  if [[ ! -z $(git diff master google-api-grpc/*${CLIENT_NAME}*) ]]; then
+    echo "true"
+    return
+  fi
+
+  echo "false"
+}
+
 # Print out Java version
 java -version
 echo $JOB_TYPE
 
+if [[ "${SKIP_INTEGRATION_TESTS_IF_NO_CHANGES}" == "true" ]] &&
+   [[ "${JOB_TYPE}" == "integration" ]]; then
+  CLIENT=$(echo ${INTEGRATION_TEST_ARGS} | cut -d' ' -f1 | cut -d'/' -f2)
+  CLIENT_HAS_CHANGES=$(client_has_changes ${CLIENT})
+  if [[ "${CLIENT_HAS_CHANGES}" == "false" ]]; then
+    echo "No difference from master, skipping tests."
+    exit 0
+  fi
+fi
+
 mvn install -DskipTests=true -Dmaven.javadoc.skip=true -Dgcloud.download.skip=true -B -V
 
 # prepend Kokoro root directory onto GOOGLE_APPLICATION_CREDENTIALS path
-if [ ! -z "$GOOGLE_APPLICATION_CREDENTIALS" ]; then
+if [[ ! -z "$GOOGLE_APPLICATION_CREDENTIALS" ]]; then
     export GOOGLE_APPLICATION_CREDENTIALS=$(realpath ${KOKORO_ROOT}/src/${GOOGLE_APPLICATION_CREDENTIALS})
 fi
 

.kokoro/common.cfg

@@ -11,3 +11,9 @@ env_vars: {
     key: "TRAMPOLINE_BUILD_FILE"
     value: "github/google-auth-library-java/.kokoro/build.sh"
 }
+
+# By default, always run integration tests
+env_vars: {
+  key: "SKIP_INTEGRATION_TESTS_IF_NO_CHANGES"
+  value: "false"
+}

.kokoro/continuous/bigtableadmin-it.cfg

@@ -8,7 +8,7 @@ env_vars: {
 
 env_vars: {
     key: "INTEGRATION_TEST_ARGS"
-    value: "google-cloud-clients/google-cloud-bigtable-admin -Dbigtable.project=gcloud-devel -Dbigtable.instance=google-cloud-bigtable"
+    value: "google-cloud-clients/google-cloud-bigtable -Dbigtable.env=prod -Dbigtable.project=gcloud-devel -Dbigtable.instance=google-cloud-bigtable -Dbigtable.table=integration-tests"
 }
 
 env_vars: {

.kokoro/nightly/bigtableadmin-it.cfg

@@ -8,7 +8,7 @@ env_vars: {
 
 env_vars: {
     key: "INTEGRATION_TEST_ARGS"
-    value: "google-cloud-clients/google-cloud-bigtable-admin -Dbigtable.project=gcloud-devel -Dbigtable.instance=google-cloud-bigtable"
+    value: "google-cloud-clients/google-cloud-bigtable -Dbigtable.env=prod -Dbigtable.project=gcloud-devel -Dbigtable.instance=google-cloud-bigtable -Dbigtable.table=integration-tests"
 }
 
 env_vars: {

.kokoro/presubmit/bigtableadmin-it.cfg

@@ -8,7 +8,7 @@ env_vars: {
 
 env_vars: {
     key: "INTEGRATION_TEST_ARGS"
-    value: "google-cloud-clients/google-cloud-bigtable-admin -Dbigtable.project=gcloud-devel -Dbigtable.instance=google-cloud-bigtable"
+    value: "google-cloud-clients/google-cloud-bigtable -Dbigtable.env=prod -Dbigtable.project=gcloud-devel -Dbigtable.instance=google-cloud-bigtable -Dbigtable.table=integration-tests"
 }
 
 env_vars: {

.kokoro/presubmit/common.cfg

@@ -14,13 +14,19 @@ gfile_resources: "/bigstore/cloud-devrel-kokoro-resources/trampoline"
 build_file: "google-cloud-java/.kokoro/trampoline.sh"
 
 env_vars: {
-    key: "TRAMPOLINE_BUILD_FILE"
-    value: "github/google-cloud-java/.kokoro/build.sh"
+  key: "TRAMPOLINE_BUILD_FILE"
+  value: "github/google-cloud-java/.kokoro/build.sh"
 }
 
 env_vars: {
-    key: "JOB_TYPE"
-    value: "test"
+  key: "JOB_TYPE"
+  value: "test"
+}
+
+# For presubmits, only run integrations if the code has changed
+env_vars: {
+  key: "SKIP_INTEGRATION_TESTS_IF_NO_CHANGES"
+  value: "true"
 }
 
 before_action {

README.md

@@ -4,7 +4,7 @@ Google Cloud Client Library for Java
 Java idiomatic client for [Google Cloud Platform][cloud-platform] services.
 
 [![Kokoro CI](http://storage.googleapis.com/cloud-devrel-public/java/badges/google-cloud-java/master.svg)](http://storage.googleapis.com/cloud-devrel-public/java/badges/google-cloud-java/master.html)
-[![Maven](https://img.shields.io/maven-central/v/com.google.cloud/google-cloud.svg)](http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22com.google.cloud%22%20a%3A%22google-cloud%22)
+[![Maven](https://img.shields.io/maven-central/v/com.google.cloud/google-cloud-bom.svg)](https://search.maven.org/search?q=g:com.google.cloud%20a:google-cloud-bom)
 [![Codacy Badge](https://api.codacy.com/project/badge/grade/9da006ad7c3a4fe1abd142e77c003917)](https://www.codacy.com/app/mziccard/google-cloud-java)
 
 - [Google Cloud Platform Documentation][cloud-platform-docs]
@@ -42,7 +42,6 @@ This library supports the following Google Cloud Platform services with clients
 This library supports the following Google Cloud Platform services with clients at an [Alpha](#versioning) quality level:
 
 -  [Cloud Bigtable](google-cloud-clients/google-cloud-bigtable) (Alpha)
--  [Cloud Bigtable Admin](google-cloud-clients/google-cloud-bigtable-admin) (Alpha)
 -  [Cloud Compute](google-cloud-clients/google-cloud-compute) (Alpha)
 -  [Cloud Dataproc](google-cloud-clients/google-cloud-dataproc) (Alpha)
 -  [Cloud DNS](google-cloud-clients/google-cloud-dns) (Alpha)
@@ -67,7 +66,7 @@ If you are using Maven, add this to your pom.xml file
       <dependency>
         <groupId>com.google.cloud</groupId>
         <artifactId>google-cloud-bom</artifactId>
-        <version>0.76.0-alpha</version>
+        <version>0.80.0-alpha</version>
         <type>pom</type>
         <scope>import</scope>
        </dependency>
@@ -86,11 +85,11 @@ If you are using Maven, add this to your pom.xml file
 [//]: # ({x-version-update-start:google-cloud-storage:released})
 If you are using Gradle, add this to your dependencies
 ```Groovy
-compile 'com.google.cloud:google-cloud-storage:1.58.0'
+compile 'com.google.cloud:google-cloud-storage:1.62.0'
 ```
 If you are using SBT, add this to your dependencies
 ```Scala
-libraryDependencies += "com.google.cloud" % "google-cloud-storage" % "1.58.0"
+libraryDependencies += "com.google.cloud" % "google-cloud-storage" % "1.62.0"
 ```
 [//]: # ({x-version-update-end})
 
@@ -111,7 +110,7 @@ If you are running into problems with version conflicts, the easiest way to solv
       <dependency>
         <groupId>com.google.cloud</groupId>
         <artifactId>google-cloud-bom</artifactId>
-        <version>0.76.0-alpha</version>
+        <version>0.80.0-alpha</version>
         <type>pom</type>
         <scope>import</scope>
       </dependency>

TESTING.md

@@ -56,8 +56,8 @@ To use the `prod` environment:
 2. Download the [JSON service account credentials file][create-service-account] from the Google 
    Developer's Console.
 3. Set the environment variable `GOOGLE_APPLICATION_CREDENTIALS` to the path of the credentials file
-4. Set the system property `bigtable.env=prod` and `bigtable.table` to the full table name you 
-    created earlier. Example: 
+4. Set the system property `bigtable.env=prod`, `bigtable.project`, `bigtable.instance` and
+   `bigtable.table` to created earlier. Example: 
     ```shell
     mvn verify -am -pl google-cloud-bigtable \
       -Dbigtable.env=prod \
@@ -66,23 +66,6 @@ To use the `prod` environment:
       -Dbigtable.table=my-table
     ```
 
-### Testing code that uses Bigtable Admin
-
-Bigtable Admin integration tests are run against a real Bigtable instance.
-
-To run the tests:
-1. Set up the target table using `google-cloud-bigtable/scripts/setup-test-table.sh`
-2. Download the [JSON service account credentials file][create-service-account] from the Google
-   Developer's Console.
-3. Set the environment variable `GOOGLE_APPLICATION_CREDENTIALS` to the path of the credentials file
-4. Set the system property `bigtable.instance` to the full instance name you
-    created earlier. Example:
-    ```shell
-    mvn verify -am -pl google-cloud-bigtable-admin \
-      -Dbigtable.project=my-project
-      -Dbigtable.instance=my-instance
-    ```
-
 ### Testing code that uses Compute
 
 Currently, there isn't an emulator for Google Compute, so an alternative is to create a test
@@ -173,50 +156,7 @@ uses the `RemoteLoggingHelper` to create a metric.
 
 You can test against a Pub/Sub emulator:
 
-1. [Install Cloud SDK](https://cloud.google.com/sdk/downloads)
-
-2. Start the emulator:
-```shell
-$ gcloud beta emulators pubsub start
-```
-
-To determine which host/port the emulator is running on:
-```shell
-$ gcloud beta emulators pubsub env-init
-# Sample output:
-#   export PUBSUB_EMULATOR_HOST=localhost:8759
-```
-
-3. Point your client to the emulator.
-```java
-String hostport = System.getenv("PUBSUB_EMULATOR_HOST");
-ManagedChannel channel = ManagedChannelBuilder.forTarget(hostport).usePlaintext(true).build();
-try {
-  ChannelProvider channelProvider = FixedChannelProvider.create(channel);
-  CredentialsProvider credentialsProvider = new NoCredentialsProvider();
-
-  // Similarly for SubscriptionAdminSettings
-  TopicAdminClient topicClient = TopicAdminClient.create(
-    TopicAdminSettings
-      .defaultBuilder()
-        .setTransportProvider(
-            GrpcTransportProvider.newBuilder()
-                .setChannelProvider(channelProvider)
-                .build())
-      .setCredentialsProvider(credentialsProvider)
-      .build());
-
-  // Similarly for Subscriber
-  Publisher publisher =
-    Publisher
-      .defaultBuilder(topicName)
-      .setChannelProvider(channelProvider)
-      .setCredentialsProvider(credentialsProvider)
-      .build();
-} finally {
-  channel.shutdown();
-}
-```
+1. [Set up the emulator](https://cloud.google.com/pubsub/docs/emulator)
 
 ### Testing code that uses Redis
 

google-api-grpc/grpc-google-cloud-asset-v1beta1/pom.xml

@@ -3,13 +3,13 @@
   xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
   <modelVersion>4.0.0</modelVersion>
   <artifactId>grpc-google-cloud-asset-v1beta1</artifactId>
-  <version>0.41.0</version><!-- {x-version-update:grpc-google-cloud-asset-v1beta1:current} -->
+  <version>0.45.1-SNAPSHOT</version><!-- {x-version-update:grpc-google-cloud-asset-v1beta1:current} -->
   <name>grpc-google-cloud-asset-v1beta1</name>
   <description>GRPC library for grpc-google-cloud-asset-v1beta1</description>
   <parent>
     <groupId>com.google.api.grpc</groupId>
     <artifactId>google-api-grpc</artifactId>
-    <version>0.41.0</version><!-- {x-version-update:google-api-grpc:current} -->
+    <version>0.45.1-SNAPSHOT</version><!-- {x-version-update:google-api-grpc:current} -->
   </parent>
   <dependencies>
     <dependency>

google-api-grpc/grpc-google-cloud-automl-v1beta1/pom.xml

@@ -3,13 +3,13 @@
   xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
   <modelVersion>4.0.0</modelVersion>
   <artifactId>grpc-google-cloud-automl-v1beta1</artifactId>
-  <version>0.41.0</version><!-- {x-version-update:grpc-google-cloud-automl-v1beta1:current} -->
+  <version>0.45.1-SNAPSHOT</version><!-- {x-version-update:grpc-google-cloud-automl-v1beta1:current} -->
   <name>grpc-google-cloud-automl-v1beta1</name>
   <description>GRPC library for grpc-google-cloud-automl-v1beta1</description>
   <parent>
     <groupId>com.google.api.grpc</groupId>
     <artifactId>google-api-grpc</artifactId>
-    <version>0.41.0</version><!-- {x-version-update:google-api-grpc:current} -->
+    <version>0.45.1-SNAPSHOT</version><!-- {x-version-update:google-api-grpc:current} -->
   </parent>
   <dependencies>
     <dependency>