Prevent Triggerer from crashing when a trigger event isn't serializable by dabla · Pull Request #60152 · apache/airflow

dabla · 2026-01-06T08:00:48Z

This PR is related to the issue #60120 in which the trigger process crashes if a returned trigger event isn't serializable. This PR will catch the NotImpelmentedError and detect which event caused the crash and cancel the trigger associated to that event that caused the error.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

…mplementedError as a trigger event is not serializable, then retry without that event and cancel associated trigger

…y happy

potiuk · 2026-01-08T13:16:12Z

Looks good and seems it would be great to have it in 3.1.6 - but I would love someone with more triggerer expertise reviews it.

…etry call are correctly asserted and bad events are being sanitized

dabla · 2026-01-19T17:12:09Z

@ash I refactored the code as discussed, first I will just try to send the events as is, if that fails, it will first sanitize the events and then retry to send it. I also adapted the test with asserts and verifications, now that scenario is fully covered.

airflow-core/src/airflow/jobs/triggerer_job_runner.py

github-actions · 2026-01-22T18:22:22Z

Backport failed to create: v3-1-test. View the failure log Run details

Status	Branch	Result
❌	v3-1-test

You can attempt to backport this manually by running:

cherry_picker 0238244 v3-1-test

This should apply the commit to the v3-1-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

…le (#60152) * refactor: If asend in TriggerRunner comms decoder crashes due to NotImplementedError as a trigger event is not serializable, then retry without that event and cancel associated trigger * refactor: Applied some reformatting * refactor: Fixed some mypy issues * refactor: Fixed return type of send_changes method * refactor: Changed imports of trigger events * refactor: Reformatted trigger job runner * refactor: Fixed mocking of comms decoder * refactor: Forgot to add the patched supervisor_builder * refactor: Changed asserts in test_sync_state_to_supervisor * refactor: Refactored how state changes are validated * refactor: Validate events while creating the TriggerStateChanges message * refactor: Refactored try/except in validate_state_changes to keep mypy happy * Update airflow-core/src/airflow/jobs/triggerer_job_runner.py Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com> * refactor: Renamed validate_state_changes method to process_trigger_events * refactor: Only sanitize invalid trigger events if first attempt fails * refactor: Should check if msg.events is not None * refactor: Adapted test_sync_state_to_supervisor so both initial and retry call are correctly asserted and bad events are being sanitized --------- Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com> (cherry picked from commit 0238244)

dabla · 2026-01-22T19:52:20Z

Backported to v3-1-test

pierrejeambrun · 2026-01-23T12:15:49Z

@dabla how did you backport this? I think it's breaking the CI and I can't find the related PR, I'm surprised this could be merged on v3-1-test without static check succeeding.

I have opened a revert PR there #60976 to see if the CI is happy.

…rializable (apache#60152)" This reverts commit b7d1c41.

…rializable (#60152)" (#60976) This reverts commit b7d1c41.

dabla · 2026-01-23T15:29:06Z

@dabla how did you backport this? I think it's breaking the CI and I can't find the related PR, I'm surprised this could be merged on v3-1-test without static check succeeding.

https://github.com/apache/airflow/actions/runs/21285259040/job/61265554224?pr=60958

https://github.com/apache/airflow/actions/runs/21285249743/job/61265446226?pr=60959

I have opened a revert PR there #60976 to see if the CI is happy.

Salut @pierrejeambrun. Well I did as was described above, I checked out the v3-1-test, cherry-picked the commit from this PR, resolved merged conflicts, and pushed it. Will create branch from v3-1-test and create PR from there, thx for reverting it.

…le (#60152) * refactor: If asend in TriggerRunner comms decoder crashes due to NotImplementedError as a trigger event is not serializable, then retry without that event and cancel associated trigger * refactor: Applied some reformatting * refactor: Fixed some mypy issues * refactor: Fixed return type of send_changes method * refactor: Changed imports of trigger events * refactor: Reformatted trigger job runner * refactor: Fixed mocking of comms decoder * refactor: Forgot to add the patched supervisor_builder * refactor: Changed asserts in test_sync_state_to_supervisor * refactor: Refactored how state changes are validated * refactor: Validate events while creating the TriggerStateChanges message * refactor: Refactored try/except in validate_state_changes to keep mypy happy * Update airflow-core/src/airflow/jobs/triggerer_job_runner.py Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com> * refactor: Renamed validate_state_changes method to process_trigger_events * refactor: Only sanitize invalid trigger events if first attempt fails * refactor: Should check if msg.events is not None * refactor: Adapted test_sync_state_to_supervisor so both initial and retry call are correctly asserted and bad events are being sanitized --------- Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com> (cherry picked from commit 0238244)

dabla · 2026-01-23T15:48:40Z

@pierrejeambrun I've create PR #60981 for the backport. Thx to @shahar1 for helping me out.

…le (#60152) * refactor: If asend in TriggerRunner comms decoder crashes due to NotImplementedError as a trigger event is not serializable, then retry without that event and cancel associated trigger * refactor: Applied some reformatting * refactor: Fixed some mypy issues * refactor: Fixed return type of send_changes method * refactor: Changed imports of trigger events * refactor: Reformatted trigger job runner * refactor: Fixed mocking of comms decoder * refactor: Forgot to add the patched supervisor_builder * refactor: Changed asserts in test_sync_state_to_supervisor * refactor: Refactored how state changes are validated * refactor: Validate events while creating the TriggerStateChanges message * refactor: Refactored try/except in validate_state_changes to keep mypy happy * Update airflow-core/src/airflow/jobs/triggerer_job_runner.py Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com> * refactor: Renamed validate_state_changes method to process_trigger_events * refactor: Only sanitize invalid trigger events if first attempt fails * refactor: Should check if msg.events is not None * refactor: Adapted test_sync_state_to_supervisor so both initial and retry call are correctly asserted and bad events are being sanitized --------- Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com> (cherry picked from commit 0238244)

…le (#60152) (#60981) * refactor: If asend in TriggerRunner comms decoder crashes due to NotImplementedError as a trigger event is not serializable, then retry without that event and cancel associated trigger * refactor: Applied some reformatting * refactor: Fixed some mypy issues * refactor: Fixed return type of send_changes method * refactor: Changed imports of trigger events * refactor: Reformatted trigger job runner * refactor: Fixed mocking of comms decoder * refactor: Forgot to add the patched supervisor_builder * refactor: Changed asserts in test_sync_state_to_supervisor * refactor: Refactored how state changes are validated * refactor: Validate events while creating the TriggerStateChanges message * refactor: Refactored try/except in validate_state_changes to keep mypy happy * Update airflow-core/src/airflow/jobs/triggerer_job_runner.py Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com> * refactor: Renamed validate_state_changes method to process_trigger_events * refactor: Only sanitize invalid trigger events if first attempt fails * refactor: Should check if msg.events is not None * refactor: Adapted test_sync_state_to_supervisor so both initial and retry call are correctly asserted and bad events are being sanitized --------- Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com> (cherry picked from commit 0238244)

dabla · 2026-01-24T19:10:10Z

PR Merged into v3-1-test.

…le (apache#60152) * refactor: If asend in TriggerRunner comms decoder crashes due to NotImplementedError as a trigger event is not serializable, then retry without that event and cancel associated trigger * refactor: Applied some reformatting * refactor: Fixed some mypy issues * refactor: Fixed return type of send_changes method * refactor: Changed imports of trigger events * refactor: Reformatted trigger job runner * refactor: Fixed mocking of comms decoder * refactor: Forgot to add the patched supervisor_builder * refactor: Changed asserts in test_sync_state_to_supervisor * refactor: Refactored how state changes are validated * refactor: Validate events while creating the TriggerStateChanges message * refactor: Refactored try/except in validate_state_changes to keep mypy happy * Update airflow-core/src/airflow/jobs/triggerer_job_runner.py Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com> * refactor: Renamed validate_state_changes method to process_trigger_events * refactor: Only sanitize invalid trigger events if first attempt fails * refactor: Should check if msg.events is not None * refactor: Adapted test_sync_state_to_supervisor so both initial and retry call are correctly asserted and bad events are being sanitized --------- Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>

refactor: If asend in TriggerRunner comms decoder crashes due to NotI…

082fc70

…mplementedError as a trigger event is not serializable, then retry without that event and cancel associated trigger

dabla requested review from dstandish and hussein-awala as code owners January 6, 2026 08:00

boring-cyborg bot added the area:Triggerer label Jan 6, 2026

Merge branch 'main' into fix/trigger-crash-serialization-comms

5ca3b39

dabla marked this pull request as draft January 6, 2026 08:12

dabla added 20 commits January 6, 2026 09:33

refactor: Applied some reformatting

b303403

refactor: Fixed some mypy issues

5de68c1

refactor: Fixed return type of send_changes method

14082fe

refactor: Changed imports of trigger events

9cbf5a8

Merge branch 'main' into fix/trigger-crash-serialization-comms

3045fad

Merge branch 'main' into fix/trigger-crash-serialization-comms

6ebf6a5

refactor: Reformatted trigger job runner

e9dc8a4

Merge branch 'main' into fix/trigger-crash-serialization-comms

65999fe

refactor: Fixed mocking of comms decoder

6d70c53

Merge branch 'main' into fix/trigger-crash-serialization-comms

693415c

refactor: Forgot to add the patched supervisor_builder

a9a6cd8

refactor: Changed asserts in test_sync_state_to_supervisor

dc581f9

Merge branch 'main' into fix/trigger-crash-serialization-comms

521aac4

Merge branch 'main' into fix/trigger-crash-serialization-comms

5b98350

refactor: Refactored how state changes are validated

60421a0

refactor: Validate events while creating the TriggerStateChanges message

60bc489

Merge branch 'main' into fix/trigger-crash-serialization-comms

fc627b0

refactor: Refactored try/except in validate_state_changes to keep myp…

a7ffaaa

…y happy

Merge branch 'main' into fix/trigger-crash-serialization-comms

0b90b78

Merge branch 'main' into fix/trigger-crash-serialization-comms

b29e947

dabla marked this pull request as ready for review January 8, 2026 13:09

potiuk approved these changes Jan 8, 2026

View reviewed changes

potiuk added this to the Airflow 3.1.6 milestone Jan 8, 2026

refactor: Adapted test_sync_state_to_supervisor so both initial and r…

4dd4649

…etry call are correctly asserted and bad events are being sanitized

dabla requested a review from ashb January 19, 2026 21:25

potiuk approved these changes Jan 20, 2026

View reviewed changes

potiuk reviewed Jan 20, 2026

View reviewed changes

airflow-core/src/airflow/jobs/triggerer_job_runner.py Show resolved Hide resolved

dabla merged commit 0238244 into apache:main Jan 22, 2026
71 checks passed

dabla mentioned this pull request Jan 22, 2026

Triggerer crashes when a SQLExecuteQueryTrigger returns a non serializable result #60120

Closed

2 tasks

pierrejeambrun added a commit to astronomer/airflow that referenced this pull request Jan 23, 2026

Revert "Prevent Triggerer from crashing when a trigger event isn't se…

facee8a

…rializable (apache#60152)" This reverts commit b7d1c41.

pierrejeambrun mentioned this pull request Jan 23, 2026

Revert "Prevent Triggerer from crashing when a trigger event isn't se… #60976

Merged

1 task

pierrejeambrun added a commit that referenced this pull request Jan 23, 2026

Revert "Prevent Triggerer from crashing when a trigger event isn't se…

937f445

…rializable (#60152)" (#60976) This reverts commit b7d1c41.

dabla mentioned this pull request Jan 23, 2026

Backport PR #60152: Prevent Triggerer from crashing when a trigger event isn't serializable #60980

Closed

1 task

ephraimbuddy added type:bug-fix Changelog: Bug Fixes changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) and removed type:bug-fix Changelog: Bug Fixes labels Jan 27, 2026

dabla mentioned this pull request Jan 28, 2026

Implement specialized get_first and get_records method in OracleHook #61144

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prevent Triggerer from crashing when a trigger event isn't serializable#60152

Prevent Triggerer from crashing when a trigger event isn't serializable#60152
dabla merged 31 commits intoapache:mainfrom
dabla:fix/trigger-crash-serialization-comms

dabla commented Jan 6, 2026

Uh oh!

potiuk commented Jan 8, 2026

Uh oh!

dabla commented Jan 19, 2026

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jan 22, 2026

Uh oh!

dabla commented Jan 22, 2026

Uh oh!

pierrejeambrun commented Jan 23, 2026 •

edited

Loading

Uh oh!

dabla commented Jan 23, 2026 •

edited

Loading

Uh oh!

dabla commented Jan 23, 2026 •

edited

Loading

Uh oh!

dabla commented Jan 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

dabla commented Jan 6, 2026

Uh oh!

potiuk commented Jan 8, 2026

Uh oh!

dabla commented Jan 19, 2026

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jan 22, 2026

Backport failed to create: v3-1-test. View the failure log Run details

Uh oh!

dabla commented Jan 22, 2026

Uh oh!

pierrejeambrun commented Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dabla commented Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dabla commented Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dabla commented Jan 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

pierrejeambrun commented Jan 23, 2026 •

edited

Loading

dabla commented Jan 23, 2026 •

edited

Loading

dabla commented Jan 23, 2026 •

edited

Loading