[DO NOT SUBMIT] Umbrella PR for setting prism as the default Go SDK runner instead of the direct runner.

[lostluck]

Umbrella PR to make prism the default runner. Will be sending out smaller slices of this separately for review.

This fixes up several paper cuts WRT the prism runner vs the direct runner in Go SDK code. As well as fixing up a few issues WRT built-ins running on portable runners.

Critically, it avoids discrepancies between the direct runner implementation and the expectations of other (portable) runners. While this does make things slightly harder to get started, it homogenizes the experience when moving to production beam runners from testing, which will make documenting correct procedures consistent, and provide consistent test implementations.

• Sets prism to be the default runner for the Go SDK.

  • Replace direct uses of direct package with prism.
  • Mark the direct runner package as Deprecated.
  • TODO: Provide clear instructions for what to do/use instead.
  • TODO: Run all the examples with prism.

• Update the SDK to require Go 1.20

  • Due to prism's exp/slog dependency, which will be migrated to the standard library version with go 1.21.
  • Going to need to make some additional repo changes to ensure this submits.

• Adds a prism stanza for the integration tests

  • TODO: Add jenkin's target and run as a precommit (like the Python Portable runs)

• Ensure all unit tested DoFns are registered. Update some things to use the generic registration package. Outstanding work to do a full conversion to the register package.

• Add calls to ptest.Main in TestMains for package unit tests that use ptest.

• The datastore and fhirio packages stay on the "direct" runner because they don't have portable testing implementations. Those issues have been filed seperately. datastore ([Task][Go SDK]: Update datastoreio tests to not rely on direct runner. #27549) and fhirio ([Task][Go SDK]: Update fhirio tests to not rely on direct runner.  #27547), This only pertains to their ability to test, not executed.

• Fixed top.accum to handle when it's not in a fused stage.

• Update the debug string for ParDo execution nodes to list side inputs.

• Update the debug string side inputs to list coder.

• Clarify harness ProcessBundle handling as the error source when the failure from materializing ProcessBundleDescriptors to exec.Plans.

• Ensure data handling errors that end up at harness are actually reported as errors that fail a bundle.

• Update the universal runner to track the last error message over the stream, and return that as the pipeline failure message. Alternatively, wait until the stream terminates or an error message is received after the JobState is Failed. This allows prism (and other portable runners) to produce the failure cause cleanly to the launching task (if it's still up).

• Have the universal runner wrapper avoid the Go binary compile step when the execution is set to Loopback mode. Generates a temporary empty file and sets that as the "worker_binary". This avoids waiting on compiles between each test, when the binary would not even be used anyway.

  • Adds "Loopback" as a field option to the runnerlib package.

• Quieted down some info logs.

• Prism Fixes

  • Ensuring coders for SideInputs are properly set if their PCollection coder is replaced. Otherwise this would lead to problems on decoding.
  • Moves test DoFns into a internal_test package. Otherwise there's a circular dependency with importing prism into ptest for executing pipelines with prism as default.
  • Track the "pipeline terminating failure" and ensure that is logged as an Error class message over the message stream.
  • Have SDK errors on Progress or Split abort the progress loop. Doesn't abort bundle processing, but prevents receiving tentative results. Prevents non-termination of prism jobs on failures, when stuck in the loop.
  • Since SDK errors are now relayed back to the launching process as log messages, no longer log them at the prism end. Removes verbose duplicate messages.

Additional TODOs

• Handle CoGBKs in prism (in particular, Expand Nodes, likely through fusion)
• Handle Reshuffles (likely as a no-op), since the Windowing Strategy/Triggers it use aren't implemented.

See #24789

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

[lostluck]

Run Go PostCommit

[lostluck]

Closing because the contents of this PR have been submitted in other PRs.