Various release script fixes (#27548)

* Various release script fixes

* Remove Release_Gradle_Build job from release verification

* Fix publish docker image script to work with remotely built images

* Change empty file to .json to make RAT_PreCommit happy

* Set base branch to the release branch for build snapshot PR

* Bump pubsub signal wait for pipeline start to 8 min in integration tests

* Change message head from Warning to Level for PubsubInjector
  as it follows by an exit(1)

* Fix Pylint

.github/workflows/cut_release_branch.yml

@@ -147,7 +147,7 @@ jobs:
             fi
           done
 
-          cat /tmp/result | sort | uniq | grep -i -E 'precommit|postcommit|validates|vr|example|test|gradle build' | grep -v -i -E 'load|perf|website' >> release/src/main/scripts/jenkins_jobs.txt
+          cat /tmp/result | sort | uniq | grep -i -E 'precommit|postcommit|validates|vr|example|test' | grep -v -i -E 'load|perf|website' >> release/src/main/scripts/jenkins_jobs.txt
         env:
           JENKINS_USERNAME: ${{ github.event.inputs.JENKINS_USERNAME }}
           JENKINS_TOKEN: ${{ github.event.inputs.JENKINS_TOKEN }}
@@ -175,6 +175,7 @@ jobs:
       REMOTE_NAME: remote_repo
       REMOTE_URL: ${{ github.server_url }}/${{ github.repository }}
       BRANCH_NAME: snapshot_build-${{ github.event.inputs.RELEASE_VERSION }}
+      RELEASE_BRANCH: release-${{ github.event.inputs.RELEASE_VERSION }}
     steps:
       - name: Install Hub
         run: |
@@ -192,15 +193,15 @@ jobs:
         run: |
           git remote add ${REMOTE_NAME} ${REMOTE_URL}
           git checkout -b ${BRANCH_NAME}
-          touch empty_file.txt
+          touch empty_file.json
           git add -A
           git commit -m "Add empty file in order to create PR"
           git push -f ${REMOTE_NAME}
       - name: Create Pull Request
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
         run: |
-          hub pull-request -F- <<<"[DO NOT MERGE]Start snapshot build for release process 
+          hub pull-request -b apache:${RELEASE_BRANCH} -F- <<<"[DO NOT MERGE]Start snapshot build for release process
           
           
           Run Gradle Publish"

.github/workflows/run_rc_validation.yml

@@ -92,8 +92,8 @@ jobs:
       - name: Create Pull Request
         run: |
           git checkout -b ${{env.WORKING_BRANCH}} ${{ env.RC_TAG }} --quiet
-          touch empty_file.txt
-          git add empty_file.txt
+          touch empty_file.json
+          git add empty_file.json
           git commit -m "Add empty file in order to create PR" --quiet
           git push origin ${{env.WORKING_BRANCH}}  --quiet
           GITHUB_PR_URL=$(gh pr create -B ${{env.RELEASE_BRANCH}} -H ${{env.WORKING_BRANCH}} -t"[DO NOT MERGE] Run Python RC Validation Tests" -b "PR to run Python ReleaseCandidate Jenkins Job.")

.test-infra/jenkins/README.md

@@ -295,7 +295,6 @@ Beam Jenkins overview page: [link](https://ci-beam.apache.org/)
 | beam_Publish_Beam_SDK_Snapshots | [cron](https://ci-beam.apache.org/job/beam_Publish_Beam_SDK_Snapshots/)| N/A | [![Build Status](https://ci-beam.apache.org/job/beam_Publish_Beam_SDK_Snapshots/badge/icon)](https://ci-beam.apache.org/job/beam_Publish_Beam_SDK_Snapshots/) |
 | beam_Publish_Docker_Snapshots | [cron](https://ci-beam.apache.org/job/beam_Publish_Docker_Snapshots/)| N/A | [![Build Status](https://ci-beam.apache.org/job/beam_Publish_Docker_Snapshots/badge/icon)](https://ci-beam.apache.org/job/beam_Publish_Docker_Snapshots/) |
 | beam_PostRelease_Python_Candidate | [cron](https://ci-beam.apache.org/job/beam_PostRelease_Python_Candidate/)| `Run Python ReleaseCandidate` | [![Build Status](https://ci-beam.apache.org/job/beam_PostRelease_Python_Candidate/badge/icon)](https://ci-beam.apache.org/job/beam_PostRelease_Python_Candidate/) |
-| beam_Release_Gradle_Build | [cron](https://ci-beam.apache.org/job/beam_Release_Gradle_Build/) | `Run Release Gradle Build` | [![Build Status](https://ci-beam.apache.org/job/beam_Release_Gradle_Build/badge/icon)](https://ci-beam.apache.org/job/beam_Release_Gradle_Build/)
 
 ### Notes:
 

examples/java/src/main/java/org/apache/beam/examples/complete/game/injector/InjectorUtils.java

@@ -47,7 +47,7 @@ public static Pubsub getClient(final HttpTransport httpTransport, final JsonFact
     }
     if (credential.getClientAuthentication() != null) {
       System.out.println(
-          "\n***Warning! You are not using service account credentials to "
+          "\n***Error! You are not using service account credentials to "
               + "authenticate.\nYou need to use service account credentials for this example,"
               + "\nsince user-level credentials do not have enough pubsub quota,\nand so you will run "
               + "out of PubSub quota very quickly.\nSee "

release/src/main/scripts/jenkins_jobs.txt

@@ -139,7 +139,6 @@ Run Python_Transforms PreCommit,beam_PreCommit_Python_Transforms_Phrase
 Run Python_Xlang_Gcp_Dataflow PostCommit,beam_PostCommit_Python_Xlang_Gcp_Dataflow_PR
 Run Python_Xlang_Gcp_Direct PostCommit,beam_PostCommit_Python_Xlang_Gcp_Direct_PR
 Run RAT PreCommit,beam_PreCommit_RAT_Phrase
-Run Release Gradle Build,beam_Release_Gradle_Build
 Run SQL PostCommit,beam_PostCommit_SQL_PR
 Run SQL PreCommit,beam_PreCommit_SQL_Phrase
 Run SQL_Java11 PreCommit,beam_PreCommit_SQL_Java11_Phrase

release/src/main/scripts/publish_docker_images.sh

@@ -37,18 +37,27 @@ echo "Which release candidate will be the source of final docker images? (ex: 1)
 read RC_NUM
 RC_VERSION="rc${RC_NUM}"
 
-echo "================Confirming Release and RC version==========="
+echo "================Pull RC Containers from DockerHub==========="
+IMAGES=$(docker search ${DOCKER_IMAGE_DEFAULT_REPO_ROOT}/${DOCKER_IMAGE_DEFAULT_REPO_PREFIX} --format "{{.Name}}" --limit 100)
+KNOWN_IMAGES=()
 echo "We are using ${RC_VERSION} to push docker images for ${RELEASE}."
+while read IMAGE; do
+  # Try pull verified RC from dockerhub.
+  if docker pull "${IMAGE}:${RELEASE}${RC_VERSION}" 2>/dev/null ; then
+    KNOWN_IMAGES+=( $IMAGE )
+  fi
+done < <(echo "${IMAGES}")
+
+echo "================Confirming Release and RC version==========="
 echo "Publishing the following images:"
-IMAGES=$(docker images --filter "reference=apache/beam_*:${RELEASE}${RC_VERSION}" --format "{{.Repository}}")
-echo "${IMAGES}"
+# Sort by name for easy examination
+IFS=$'\n' KNOWN_IMAGES=($(sort <<<"${KNOWN_IMAGES[*]}"))
+unset IFS
+printf "%s\n" ${KNOWN_IMAGES[@]}
 echo "Do you want to proceed? [y|N]"
 read confirmation
 if [[ $confirmation = "y" ]]; then
-  echo "${IMAGES}" | while read IMAGE; do
-    # Pull verified RC from dockerhub.
-    docker pull "${IMAGE}:${RELEASE}${RC_VERSION}"
-
+  for IMAGE in "${KNOWN_IMAGES[@]}"; do
     # Tag with ${RELEASE} and push to dockerhub.
     docker tag "${IMAGE}:${RELEASE}${RC_VERSION}" "${IMAGE}:${RELEASE}"
     docker push "${IMAGE}:${RELEASE}"
@@ -58,4 +67,4 @@ if [[ $confirmation = "y" ]]; then
     docker push "${IMAGE}:latest"
   done
 
-fi
+fi

release/src/main/scripts/run_rc_validation.sh

@@ -271,8 +271,8 @@ echo "This task will create a PR against apache/beam, trigger a jenkins job to r
 echo "1. Python quickstart validations(batch & streaming)"
 echo "2. Python MobileGame validations(UserScore, HourlyTeamScore)"
 if [[ "$python_quickstart_mobile_game" = true && ! -z `which hub` ]]; then
-  touch empty_file.txt
-  git add empty_file.txt
+  touch empty_file.json
+  git add empty_file.json
   git commit -m "Add empty file in order to create PR" --quiet
   git push -f ${GITHUB_USERNAME} --quiet
   # Create a test PR

release/src/main/scripts/verify_release_build.sh

@@ -125,8 +125,8 @@ hub version
 
 
 echo ""
-echo "==================== 3 Run Gradle Release Build & PostCommit Tests on Jenkins ==================="
-echo "[Current Task] Run Gradle release build and all PostCommit Tests against Release Branch on Jenkins."
+echo "==================== 3 Run PostCommit Tests on Jenkins ==================="
+echo "[Current Task] Run all PostCommit Tests against Release Branch on Jenkins."
 echo "This task will create a PR against apache/beam."
 echo "After PR created, you need to comment phrases listed in description in the created PR:"
 

sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/healthcare/FhirIOReadIT.java

@@ -133,7 +133,7 @@ public void testFhirIORead() throws Exception {
         "waitForAnyMessage", signal.signalSuccessWhen(resources.getCoder(), anyResources -> true));
     // wait for any resource
 
-    Supplier<Void> start = signal.waitForStart(Duration.standardMinutes(5));
+    Supplier<Void> start = signal.waitForStart(Duration.standardMinutes(8));
     pipeline.apply(signal.signalStart());
     PipelineResult job = pipeline.run();
     start.get();

sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubReadIT.java

@@ -51,7 +51,7 @@ public void testReadPublicData() throws Exception {
     messages.apply(
         "waitForAnyMessage", signal.signalSuccessWhen(messages.getCoder(), anyMessages -> true));
 
-    Supplier<Void> start = signal.waitForStart(Duration.standardMinutes(5));
+    Supplier<Void> start = signal.waitForStart(Duration.standardMinutes(8));
     pipeline.apply(signal.signalStart());
     PipelineResult job = pipeline.run();
     start.get();
@@ -79,7 +79,7 @@ public void testReadPubsubMessageId() throws Exception {
         "isMessageIdNonNull",
         signal.signalSuccessWhen(messages.getCoder(), new NonEmptyMessageIdCheck()));
 
-    Supplier<Void> start = signal.waitForStart(Duration.standardMinutes(5));
+    Supplier<Void> start = signal.waitForStart(Duration.standardMinutes(8));
     pipeline.apply(signal.signalStart());
     PipelineResult job = pipeline.run();
     start.get();

sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsublite/ReadWriteIT.java

@@ -318,7 +318,7 @@ public void testPubsubLiteWriteReadWithSchemaTransform() throws Exception {
                       return Objects.requireNonNull(row.getInt64("numberInInt")).intValue();
                     }));
     ids.apply("PubsubSignalTest", signal.signalSuccessWhen(BigEndianIntegerCoder.of(), testIds()));
-    Supplier<Void> start = signal.waitForStart(Duration.standardMinutes(5));
+    Supplier<Void> start = signal.waitForStart(Duration.standardMinutes(8));
     pipeline.apply("start signal", signal.signalStart());
     PipelineResult job = pipeline.run();
     start.get();
@@ -363,7 +363,7 @@ public void testReadWrite() throws Exception {
     PCollection<SequencedMessage> messages = readMessages(subscription, pipeline);
     PCollection<Integer> ids = messages.apply(MapElements.via(extractIds()));
     ids.apply("PubsubSignalTest", signal.signalSuccessWhen(BigEndianIntegerCoder.of(), testIds()));
-    Supplier<Void> start = signal.waitForStart(Duration.standardMinutes(5));
+    Supplier<Void> start = signal.waitForStart(Duration.standardMinutes(8));
     pipeline.apply(signal.signalStart());
     PipelineResult job = pipeline.run();
     start.get();

sdks/python/apache_beam/runners/worker/sdk_worker_test.py

@@ -23,7 +23,6 @@
 import logging
 import unittest
 from collections import namedtuple
-from typing import Any
 
 import grpc
 import hamcrest as hc

website/www/site/content/en/contribute/release-guide.md

@@ -318,7 +318,7 @@ There are 2 ways to perform this verification, either running automation script(
      ```
      cd beam/release/src/main/scripts && ./verify_release_build.sh
      ```
-  1. Trigger `beam_Release_Gradle_Build` and all Jenkins PostCommit jobs from the PR created by the previous step.
+  1. Trigger all Jenkins PostCommit jobs from the PR created by the previous step.
      You can run [mass_comment.py](https://github.com/apache/beam/blob/master/release/src/main/scripts/mass_comment.py) to do that.
      Or manually add one trigger phrase per PR comment.
      See [jenkins_jobs.txt](https://github.com/apache/beam/blob/master/release/src/main/scripts/jenkins_jobs.txt)
@@ -328,9 +328,6 @@ There are 2 ways to perform this verification, either running automation script(
   1. Installs ```hub``` with your agreement and setup local git repo;
   1. Create a test PR against release branch;
 
-The [`beam_Release_Gradle_Build`](https://ci-beam.apache.org/job/beam_Release_Gradle_Build/) Jenkins job runs `./gradlew build -PisRelease`.
-This only verifies that everything builds with unit tests passing.
-
 #### Verify the build succeeds
 
 * Tasks you need to do manually to __verify the build succeed__: