Skip to content

Conversation

@gopidesupavan
Copy link
Member

closes #47854

 [2025-03-14, 14:56:50] INFO -  Previous state of the Task instance: queued chan="stdout" source="task"
[2025-03-14, 14:56:50] INFO - Current task name:branching_ext_python chan="stdout" source="task"
[2025-03-14, 14:56:50] INFO - Dag name:example_branch_operator chan="stdout" source="task"
[2025-03-14, 14:56:50] INFO - Use 'pickle' as serializer. source="airflow.task.operators.airflow.providers.standard.operators.python.BranchExternalPythonOperator"
[2025-03-14, 14:56:50] INFO - Executing cmd: /usr/local/bin/python /tmp/venv-callo4834r1x/script.py /tmp/venv-callo4834r1x/script.in /tmp/venv-callo4834r1x/script.out /tmp/venv-callo4834r1x/string_args.txt /tmp/venv-callo4834r1x/termination.log /tmp/venv-callo4834r1x/airflow_context.json source="airflow.utils.process_utils"
[2025-03-14, 14:56:50] INFO - Output: source="airflow.utils.process_utils"
[2025-03-14, 14:56:51] INFO - Traceback (most recent call last): source="airflow.utils.process_utils"
[2025-03-14, 14:56:51] INFO -   File "/tmp/venv-callo4834r1x/script.py", line 10, in <module> source"airflow.utils.process_utils"
[2025-03-14, 14:56:51] INFO -     from airflow.plugins_manager import integrate_macros_plugins source="airflow.utils.process_utils"
[2025-03-14, 14:56:51] INFO -   File "/opt/airflow/airflow/__init__.py", line 78, in <module> source="airflow.utils.process_utils"
[2025-03-14, 14:56:51] INFO -     settings.initialize() source="airflow.utils.process_utils"
[2025-03-14, 14:56:51] INFO -   File "/opt/airflow/airflow/settings.py", line 638, in initialize source="airflow.utils.process_utils"
[2025-03-14, 14:56:51] INFO -     configure_orm() source="airflow.utils.process_utils"
[2025-03-14, 14:56:51] INFO -   File "/opt/airflow/airflow/settings.py", line 363, in configure_orm source="airflow.utils.process_utils"
[2025-03-14, 14:56:51] INFO -     engine = create_engine(SQL_ALCHEMY_CONN, connect_args=connect_args, **engine_args, future=True) source="airflow.utils.process_utils"
[2025-03-14, 14:56:51] INFO -   File "<string>", line 2, in create_engine source="airflow.utils.process_utils"
[2025-03-14, 14:56:51] INFO -   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/deprecations.py", line 375, in warned source="airflow.utils.process_utils"
[2025-03-14, 14:56:51] INFO -     return fn(*args, **kwargs) source="airflow.utils.process_utils"
[2025-03-14, 14:56:51] INFO -   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 514, in create_engine source="airflow.utils.process_utils"
[2025-03-14, 14:56:51] INFO -     u = _url.make_url(url) source="airflow.utils.process_utils"
[2025-03-14, 14:56:51] INFO -   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/url.py", line 738, in make_url source="airflow.utils.process_utils"
[2025-03-14, 14:56:51] INFO -     return _parse_url(name_or_url) source="airflow.utils.process_utils"
[2025-03-14, 14:56:51] INFO -   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/url.py", line 799, in _parse_url source="airflow.utils.process_utils"
[2025-03-14, 14:56:51] INFO -     raise exc.ArgumentError( source="airflow.utils.process_utils"
[2025-03-14, 14:56:51] INFO - sqlalchemy.exc.ArgumentError: Could not parse SQLAlchemy URL from string 'airflow-db-not-allowed:///' source="airflow.utils.process_utils"
[2025-03-14, 14:56:51] ERROR - Task failed with exception source="task" error_detail=[{"exc_type":"CalledProcessError","exc_value":"Command '['/usr/local/bin/python', '/tmp/venv-callo4834r1x/script.py', '/tmp/venv-callo4834r1x/script.in', '/tmp/venv-callo4834r1x/script.out', '/tmp/venv-callo4834r1x/string_args.txt', '/tmp/venv-callo4834r1x/termination.log', '/tmp/venv-callo4834r1x/airflow_context.json']' returned non-zero exit status 1.","exc_notes":[],"syntax_error":null,"is_cause":false,"frames":[{"filename":"/opt/airflow/task-sdk/src/airflow/sdk/execution_time/task_runner.py","lineno":608,"name":"run"},{"filename":"/opt/airflow/task-sdk/src/airflow/sdk/execution_time/task_runner.py","lineno":744,"name":"_execute_task"},{"filename":"/opt/airflow/task-sdk/src/airflow/sdk/definitions/baseoperator.py","lineno":373,"name":"wrapper"},{"filename":"/opt/airflow/providers/standard/src/airflow/providers/standard/operators/python.py","lineno":1090,"name":"execute"},{"filename":"/opt/airflow/task-sdk/src/airflow/sdk/definitions/baseoperator.py","lineno":373,"name":"wrapper"},{"filename":"/opt/airflow/providers/standard/src/airflow/providers/standard/operators/python.py","lineno":479,"name":"execute"},{"filename":"/opt/airflow/task-sdk/src/airflow/sdk/definitions/baseoperator.py","lineno":373,"name":"wrapper"},{"filename":"/opt/airflow/providers/standard/src/airflow/providers/standard/operators/python.py","lineno":201,"name":"execute"},{"filename":"/opt/airflow/providers/standard/src/airflow/providers/standard/operators/python.py","lineno":998,"name":"execute_callable"},{"filename":"/opt/airflow/providers/standard/src/airflow/providers/standard/operators/python.py","lineno":555,"name":"_execute_python_callable_in_subprocess"},{"filename":"/opt/airflow/airflow/utils/process_utils.py","lineno":175,"name":"execute_in_subprocess"},{"filename":"/opt/airflow/airflow/utils/process_utils.py","lineno":198,"name":"execute_in_subprocess_with_kwargs"}]}]
[2025-03-14, 14:56:51] INFO - Task instance in failure state chan="stdout" source="task"

The error previously was getting, inside virtual env its trying to initialise the configure_orm , causing this no db access exception.

image image

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@gopidesupavan
Copy link
Member Author

If there are any other ideas please suggest, will apply it. :)

@gopidesupavan
Copy link
Member Author

I thought of moving integrate_macros_plugins to task_sdk, i think it requires refactoring into plugin_manager. should i move it in separate PR?

@gopidesupavan gopidesupavan changed the title Fix python operators errors when initialising plugins Fix python operators errors when initialising plugins in virtualenv jinja script Mar 20, 2025
@gopidesupavan
Copy link
Member Author

ooo some docs issues , i thinks its not finding some doc reference.

@gopidesupavan
Copy link
Member Author

Merging now docs failure not related to this , its intermediate issue.

@gopidesupavan gopidesupavan merged commit 23260e4 into apache:main Mar 21, 2025
43 of 44 checks passed
@gopidesupavan gopidesupavan deleted the fix-branch-python branch March 21, 2025 22:01
agupta01 pushed a commit to agupta01/airflow that referenced this pull request Mar 21, 2025
…inja script (apache#48035)

* Fix python operators errors when initializing plugins

* Update reason for setting the PYTHON_OPERATORS_VIRTUAL_ENV_MODE environment variable in jinja
shubham-pyc pushed a commit to shubham-pyc/airflow that referenced this pull request Mar 22, 2025
…inja script (apache#48035)

* Fix python operators errors when initializing plugins

* Update reason for setting the PYTHON_OPERATORS_VIRTUAL_ENV_MODE environment variable in jinja
nailo2c pushed a commit to nailo2c/airflow that referenced this pull request Apr 4, 2025
…inja script (apache#48035)

* Fix python operators errors when initializing plugins

* Update reason for setting the PYTHON_OPERATORS_VIRTUAL_ENV_MODE environment variable in jinja
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Get BranchExternalPythonOperator with Task SDK

2 participants