Fix DAG bundle imports in subprocess operators #57631

gfreitash · 2025-10-31T16:03:30Z

closes: #56783

Resolve ModuleNotFoundError for DAG bundle imports within standard operator subprocesses in Airflow 3.x.

Problem

Subprocesses spawned by these operators don't have the DAG bundle path in PYTHONPATH (see #53617), causing imports to fail when modules from the bundle are imported inside the callable function.

Solution

Extracts bundle path from task instance context during execution call
Add extracted bundle path to subprocess PYTHONPATH environment variable
Guard with AIRFLOW_V_3_0_PLUS check for backward compatibility

boring-cyborg · 2025-10-31T16:03:33Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

potiuk

It looks great. I would love however that someone else from the bundle-work-related commiters take a look.

@ephraimbuddy @jedcunningham ? Maybe one of you :) ?

providers/standard/tests/unit/standard/operators/test_python.py

jason810496

should I mark the reviews as resolved myself or the reviewer or other maintained will do that?

Thanks for the update! Please feel free to resolve the addressed comments.

Lee-W

Looks good. Just rebased. let's see whether there's any other failure

gfreitash · 2025-11-26T17:17:08Z

By rebasing I have overwritten the previous workflow Lee-W was waiting for. Here is the run link: https://github.com/apache/airflow/actions/runs/19510952139?pr=57631

On the latest rebase the same issue of "Socket operation on non socket" persists on my local tests, so I don't think it is relevant to run the CI again for now, given the time it takes

gfreitash · 2025-12-11T11:07:09Z

Changed the test to mock subprocess execution and just test if the bundle path is in the env argument passed to the subprocess. This avoid the issue of Socket operation on non-socket on the ExternalPythonOperator that only happens in the testing environment

Add the bundle path to PYTHONPATH in operators on the standard provider that runs in a subprocess to fix import errors in Airflow 3.x.

The _callable_that_imports_from_bundle function was catching broad Exception which could mask unexpected errors. Changed to catch ImportError specifically since that's what we're testing for.

Move DAG bundle import tests from mixin pattern to a dedicated parameterized test class to improve test isolation. The new TestDagBundleImportInSubprocess class tests subprocess-based Python operators using pytest parametrization.

… of plain string on the first argument

- Fix Dag nomenclature usage - Simplify bundle path obtention logic - Add bunde path after user-defined paths - Use tmp_path pytest fixture instead of a manual one

- Return `None` if `bundle_instance` is not present to avoid attribute errors. - Ensure the bundle path is only added to `PYTHONPATH` when defined

- Mock subprocess execution to avoid environment-related issues. - Test if the bundle path was passed to the subprocess in the env argument instead

jscheffl

Looks good in my view.

* Fix DAG bundle imports in subprocess operators (apache#56783) Add the bundle path to PYTHONPATH in operators on the standard provider that runs in a subprocess to fix import errors in Airflow 3.x. * Apply precommit formatting changes * Change broad exception to specific ImportError in DAG bundle test helper The _callable_that_imports_from_bundle function was catching broad Exception which could mask unexpected errors. Changed to catch ImportError specifically since that's what we're testing for. * Refactor DAG bundle import tests into separate parameterized test class Move DAG bundle import tests from mixin pattern to a dedicated parameterized test class to improve test isolation. The new TestDagBundleImportInSubprocess class tests subprocess-based Python operators using pytest parametrization. * Fix CI ruff static check, parametrize attribute expects tuple instead of plain string on the first argument * Apply review suggestions - Fix Dag nomenclature usage - Simplify bundle path obtention logic - Add bunde path after user-defined paths - Use tmp_path pytest fixture instead of a manual one * Fix bundle path handling in subprocess operators - Return `None` if `bundle_instance` is not present to avoid attribute errors. - Ensure the bundle path is only added to `PYTHONPATH` when defined * Refactor DAG bundle import test - Mock subprocess execution to avoid environment-related issues. - Test if the bundle path was passed to the subprocess in the env argument instead * Remove unused pytest marks.

boring-cyborg bot added area:providers provider:standard labels Oct 31, 2025

gfreitash marked this pull request as draft October 31, 2025 16:37

gfreitash marked this pull request as ready for review October 31, 2025 16:45

eladkal requested a review from potiuk November 1, 2025 07:57

potiuk force-pushed the fix-56783-standard-operators-import branch from 931175e to b582f37 Compare November 1, 2025 18:16

ephraimbuddy force-pushed the v3-1-test branch from 213c76c to ea401c8 Compare November 3, 2025 15:51

ephraimbuddy requested review from XD-DENG, amoghrajesh, ashb, bugraoz93, dstandish, eladkal, ephraimbuddy, gopidesupavan, hussein-awala, jason810496, jedcunningham, jscheffl, kaxil, o-nikolas, pierrejeambrun, rawwar, shubhamraj-git and vincbeck as code owners November 3, 2025 15:51

ephraimbuddy force-pushed the v3-1-test branch from 3d1ae9f to 65d47a1 Compare November 4, 2025 09:13

potiuk approved these changes Nov 4, 2025

View reviewed changes

ephraimbuddy reviewed Nov 6, 2025

View reviewed changes

providers/standard/tests/unit/standard/operators/test_python.py Outdated Show resolved Hide resolved

ephraimbuddy reviewed Nov 6, 2025

View reviewed changes

providers/standard/tests/unit/standard/operators/test_python.py Outdated Show resolved Hide resolved

gfreitash force-pushed the fix-56783-standard-operators-import branch from e350ea6 to 0c56586 Compare November 17, 2025 17:38

jason810496 reviewed Nov 17, 2025

View reviewed changes

gfreitash force-pushed the fix-56783-standard-operators-import branch 2 times, most recently from 6281cbd to 35200dd Compare November 18, 2025 11:55

Lee-W force-pushed the fix-56783-standard-operators-import branch from 35200dd to b4d9078 Compare November 19, 2025 10:36

Lee-W approved these changes Nov 19, 2025

View reviewed changes

gfreitash force-pushed the fix-56783-standard-operators-import branch from 9472f43 to ee0a5eb Compare November 26, 2025 14:27

gfreitash force-pushed the fix-56783-standard-operators-import branch from ee0a5eb to f7536af Compare December 11, 2025 11:05

gfreitash requested review from Lee-W, ashb, ephraimbuddy and jason810496 December 12, 2025 11:43

gfreitash added 9 commits December 23, 2025 19:04

Fix DAG bundle imports in subprocess operators (apache#56783)

df15cf7

Add the bundle path to PYTHONPATH in operators on the standard provider that runs in a subprocess to fix import errors in Airflow 3.x.

Apply precommit formatting changes

587f162

Change broad exception to specific ImportError in DAG bundle test helper

f7e3bcd

The _callable_that_imports_from_bundle function was catching broad Exception which could mask unexpected errors. Changed to catch ImportError specifically since that's what we're testing for.

Fix CI ruff static check, parametrize attribute expects tuple instead…

52b6d46

… of plain string on the first argument

Apply review suggestions

5f87f79

- Fix Dag nomenclature usage - Simplify bundle path obtention logic - Add bunde path after user-defined paths - Use tmp_path pytest fixture instead of a manual one

Fix bundle path handling in subprocess operators

52eca2b

- Return `None` if `bundle_instance` is not present to avoid attribute errors. - Ensure the bundle path is only added to `PYTHONPATH` when defined

Refactor DAG bundle import test

0d4bc42

- Mock subprocess execution to avoid environment-related issues. - Test if the bundle path was passed to the subprocess in the env argument instead

Remove unused pytest marks.

f6e8d68

gfreitash force-pushed the fix-56783-standard-operators-import branch from d3692ab to f6e8d68 Compare December 23, 2025 22:10

Lee-W approved these changes Dec 24, 2025

View reviewed changes

jscheffl approved these changes Dec 24, 2025

View reviewed changes

jscheffl merged commit e2e286d into apache:main Dec 24, 2025
126 checks passed

shahar1 mentioned this pull request Dec 30, 2025

Status of testing Providers that were prepared on December 30, 2025 #59952

Closed

54 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix DAG bundle imports in subprocess operators #57631

Fix DAG bundle imports in subprocess operators #57631

Uh oh!

gfreitash commented Oct 31, 2025 •

edited

Loading

Uh oh!

boring-cyborg bot commented Oct 31, 2025

Uh oh!

potiuk left a comment

Uh oh!

Uh oh!

Uh oh!

jason810496 left a comment •

edited

Loading

Uh oh!

Lee-W left a comment

Uh oh!

gfreitash commented Nov 26, 2025

Uh oh!

gfreitash commented Dec 11, 2025

Uh oh!

jscheffl left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Fix DAG bundle imports in subprocess operators #57631

Fix DAG bundle imports in subprocess operators #57631

Uh oh!

Conversation

gfreitash commented Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Solution

Uh oh!

boring-cyborg bot commented Oct 31, 2025

Uh oh!

potiuk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jason810496 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Lee-W left a comment

Choose a reason for hiding this comment

Uh oh!

gfreitash commented Nov 26, 2025

Uh oh!

gfreitash commented Dec 11, 2025

Uh oh!

jscheffl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

gfreitash commented Oct 31, 2025 •

edited

Loading

jason810496 left a comment •

edited

Loading