Enable prism by default (where supported) #34612

damccorm · 2025-04-11T19:52:12Z

Changes the default Python runner to be prism (when prism is supported). Excludes it from interactive environments for now.

There are a bunch of use cases automatically excluded, but this also required updating a bunch of tests. The 2 main breaking changes this introduces are:

Instead of raising Python-specific exceptions, when Prism jobs fail a RuntimeError is always raised. To fix any tests relying on the old error, change your exception catching logic to look for broader errors and use regexes to verify the contents of the error. There are lots of examples of this in this PR.
Prism names its metrics differently than the local runner. For this PR in Beam, I just kept all of those on the old FnApiRunner for now, but you could also just change the metric name you're looking for (this will be the eventual fix for these tests in this repo).

If this change breaks you in unexpected ways, please:

Pin to the FnApiRunner instead of the DirectRunner. There are lots of examples of this in this PR.
Comment in [Task]: Make Prism default #34549 describing your broken use case.

Part of #34549

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

…mccorm/prismByDefault

damccorm · 2025-05-12T17:25:43Z

Looks like

beam/sdks/python/apache_beam/transforms/ptransform.py

Line 579 in 0c92ee9

def __ror__(self, left, label=None):

is the source of a lot of the test failures and that we're not handling applying transforms to non-PCollections in a portable-friendly way. This currently keys off of the process id and pipeline id (not-pickle safe), which is probably causing some problems -

beam/sdks/python/apache_beam/transforms/ptransform.py

Line 152 in 0c92ee9

def _allocate_materialized_pipeline(pipeline):

EDIT: I think that this is actually a function of pipelines double executing before #34921 was patched. So I think this is fine and my temporary patch can be removed

damccorm · 2025-05-13T19:28:20Z

sdks/python/apache_beam/options/pipeline_options.py

@@ -405,7 +405,7 @@ def get_all_options(
    # sub-classes in the main session might be repeated. Pick last unique
    # instance of each subclass to avoid conflicts.
    subset = {}
-    parser = _BeamArgumentParser()
+    parser = _BeamArgumentParser(allow_abbrev=False)


This was causing all sorts of problems because --output was getting assigned to --output_executable_path (https://github.com/apache/beam/blob/f6d209ff6a3ea638f11dfb96e52f7c21292dfbc1/sdks/python/apache_beam/options/pipeline_options.py#L1693C12-L1693C34) which meant the portable runner couldn't find prism

…mccorm/prismByDefault

damccorm · 2025-07-01T16:32:47Z

R: @shunping

github-actions · 2025-07-01T16:33:53Z

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

sdks/python/apache_beam/io/fileio_test.py

shunping · 2025-07-14T14:16:28Z

sdks/python/apache_beam/ml/inference/tensorflow_inference_test.py

@@ -128,7 +128,8 @@ def test_predict_tensor_with_batch_size(self):
    model = _create_mult2_model()
    model_path = os.path.join(self.tmpdir, f'mult2_{uuid.uuid4()}.keras')
    tf.keras.models.save_model(model, model_path)
-    with TestPipeline() as pipeline:
+    # FnApiRunner guarantees large batches, which this pipeline assumes


This could be due to the fact that prism is using accelerated processing time by default. The processing-time timer in BatchElements may not work as expected.

PR #35202 added an experimental flag for using real-time processing time, which may help to resolve this problem.

shunping · 2025-07-14T14:20:59Z

sdks/python/apache_beam/runners/direct/direct_runner.py

+        # more targeted, but for now we'll just ignore all unsafe triggers.
+        if pipeline.allow_unsafe_triggers:
+          self.supported_by_prism_runner = False
+        # TODO(https://github.com/apache/beam/issues/33623): Prism currently


What else problems do you face here when using prism in interactive mode?

sdks/python/apache_beam/runners/direct/direct_runner.py

shunping · 2025-07-14T14:33:33Z

sdks/python/apache_beam/transforms/util_test.py

@@ -254,8 +254,8 @@ def test_constant_batch_no_metrics(self):
    self.assertEqual(len(results["distributions"]), 0)

  def test_grows_to_max_batch(self):
-    # Assumes a single bundle...
-    with TestPipeline() as p:
+    # Assumes a single bundle, so we pin to the FnApiRunner


Do we have a reliable way of getting elements in a single bundle? Maybe only for testing purpose. This seems to be a recurrent issue I meet when I am working on some other tests.

Alternatively, we may be able to change the tests so that they won't rely on this assumption, but it is not working all the time.

shunping

Thank you for making this big step for prism! I really appreciate the thorough and detailed work you've put into this.

The overall PR looks solid. I've left a few minor comments for your consideration.

damccorm · 2025-07-15T13:36:17Z

There were failing postcommit tests, but not on prism, so I'm not going to block merging

Abacn · 2025-07-16T14:23:16Z

There are tests failing after this PR, could it be related?

https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Xlang_Gcp_Direct.yml?query=event%3Aschedule

https://github.com/apache/beam/actions/workflows/beam_PostCommit_Yaml_Xlang_Direct.yml?query=event%3Aschedule

damccorm · 2025-07-16T17:30:31Z

There are tests failing after this PR, could it be related?

https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Xlang_Gcp_Direct.yml?query=event%3Aschedule

https://github.com/apache/beam/actions/workflows/beam_PostCommit_Yaml_Xlang_Direct.yml?query=event%3Aschedule

I think it likely is - #35608 tested these workflows with extended timeouts and they still failed. Temporarily reverting the core of this change here - #35610

Enable prism by default

4d9c497

github-actions bot added python runners labels Apr 11, 2025

damccorm added 9 commits April 11, 2025 16:16

Clean up fallback code

99837ca

Clean up fallback code

e28e463

Add missing base case

837ba2f

Add missing base case

3233b69

Fix tests

4929906

Fix tests

ad9b0dc

exclude unsupported state

20370cb

Merge branch 'master' of https://github.com/apache/beam into users/da…

cbaa083

…mccorm/prismByDefault

Merge branch 'master' of https://github.com/apache/beam into users/da…

7e13214

…mccorm/prismByDefault

damccorm added 2 commits May 12, 2025 14:11

Exclude locally materialized results

a6db1a4

Fix fallback/dupe execution

1e49f55

This was referenced May 12, 2025

Attempt prism for pipelines with unbounded PCollections. #34721

Closed

Correctly auto-enable prism for some pipelines #34921

Merged

Fix some snippet tests

e6ca143

github-actions bot added the examples label May 13, 2025

Disable argparse abbreviation

4e3a675

damccorm commented May 13, 2025

View reviewed changes

Some test fixes for prism switch

a9e07a7

github-actions bot added the io label May 14, 2025

damccorm added 3 commits May 14, 2025 15:02

Merge in master

94ad26a

lint

bda68b1

row fixes

d8b3534

github-actions bot added the yaml label May 15, 2025

Make row comp logic default

c21dafa

github-actions bot removed the yaml label May 15, 2025

Add more yaml examples

b8343e6

minor cleanup

8a8ed63

github-actions bot removed the prism label Jun 27, 2025

damccorm added 7 commits June 27, 2025 10:59

Some more minor fixes

83472ce

Merge branch 'master' of https://github.com/apache/beam into users/da…

6efda27

…mccorm/prismByDefault

Merge in master

8fde571

Fix test

068d3a0

Fix another test with questionable assumptions

a5dde5b

Dont wait on tmpfile being destroyed

cb97e70

Merge branch 'master' of https://github.com/apache/beam into users/da…

3644d18

…mccorm/prismByDefault

damccorm marked this pull request as ready for review July 1, 2025 16:32

shunping reviewed Jul 14, 2025

View reviewed changes

sdks/python/apache_beam/io/fileio_test.py Outdated Show resolved Hide resolved

shunping reviewed Jul 14, 2025

View reviewed changes

sdks/python/apache_beam/runners/direct/direct_runner.py Show resolved Hide resolved

shunping reviewed Jul 14, 2025

View reviewed changes

shunping approved these changes Jul 14, 2025

View reviewed changes

damccorm added 6 commits July 14, 2025 10:56

More regex checks

9e2eb68

Merge in master

ae1054f

Add more postcommits, clean up some error messages

83ec444

Clean up a few tests

c1e75b3

simplify test

fa1e54d

Clean up test errors

9385804

damccorm merged commit f7071ef into master Jul 15, 2025
108 of 122 checks passed

damccorm deleted the users/damccorm/prismByDefault branch July 15, 2025 13:36

damccorm mentioned this pull request Jul 16, 2025

Temporarily move back to fnapi runner as default #35610

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Enable prism by default (where supported) #34612

Enable prism by default (where supported) #34612

Uh oh!

damccorm commented Apr 11, 2025 •

edited

Loading

Uh oh!

damccorm commented May 12, 2025 •

edited

Loading

Uh oh!

damccorm May 13, 2025 •

edited

Loading

Uh oh!

damccorm commented Jul 1, 2025

Uh oh!

github-actions bot commented Jul 1, 2025

Uh oh!

Uh oh!

shunping Jul 14, 2025 •

edited

Loading

Uh oh!

shunping Jul 14, 2025 •

edited

Loading

Uh oh!

Uh oh!

shunping Jul 14, 2025 •

edited

Loading

Uh oh!

shunping left a comment

Uh oh!

damccorm commented Jul 15, 2025 •

edited

Loading

Uh oh!

Uh oh!

Abacn commented Jul 16, 2025

Uh oh!

damccorm commented Jul 16, 2025

Uh oh!

Uh oh!

Enable prism by default (where supported) #34612

Enable prism by default (where supported) #34612

Uh oh!

Conversation

damccorm commented Apr 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

GitHub Actions Tests Status (on master branch)

Uh oh!

damccorm commented May 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

damccorm May 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

damccorm commented Jul 1, 2025

Uh oh!

github-actions bot commented Jul 1, 2025

Uh oh!

Uh oh!

shunping Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shunping Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

shunping Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shunping left a comment

Choose a reason for hiding this comment

Uh oh!

damccorm commented Jul 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Abacn commented Jul 16, 2025

Uh oh!

damccorm commented Jul 16, 2025

Uh oh!

Uh oh!

damccorm commented Apr 11, 2025 •

edited

Loading

damccorm commented May 12, 2025 •

edited

Loading

damccorm May 13, 2025 •

edited

Loading

shunping Jul 14, 2025 •

edited

Loading

shunping Jul 14, 2025 •

edited

Loading

shunping Jul 14, 2025 •

edited

Loading

damccorm commented Jul 15, 2025 •

edited

Loading