Skip to content

Conversation

@phanikumv
Copy link
Contributor

@phanikumv phanikumv commented Mar 23, 2023

This PR fixes the below error on AzureDataFactoryRunPipelineOperator when both the deferrable param and wait for termination param are set to True:

[2023-03-23, 06:12:29 UTC] {taskinstance.py:1844} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/opt/airflow/airflow/providers/microsoft/azure/operators/data_factory.py", line 225, in execute_complete
    raise AirflowException(event["message"])
airflow.exceptions.AirflowException: 'NoneType' object has no attribute 'request'

cc @josh-fell


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@josh-fell
Copy link
Contributor

@phanikumv Would you mind elaborating on how this fixes the error you noticed? It's not entirely obvious to me.

@phanikumv
Copy link
Contributor Author

@phanikumv Would you mind elaborating on how this fixes the error you noticed? It's not entirely obvious to me.

It's similar to the issue mentioned here odrling/peony-twitter#41 , the client session if initialized in context manager doesnt work as expected as the asyncio session object is getting closed before the task gets completed. The context manager exit is called before code in the body.

When I move the client creation outside context manager - it works flawlessly. PFB a screenshot of successful task runs after this change

image

@josh-fell
Copy link
Contributor

Thanks for the extra context @phanikumv!

@josh-fell josh-fell merged commit 0e69ca7 into apache:main Mar 29, 2023
josh-fell added a commit to josh-fell/airflow that referenced this pull request Mar 29, 2023
As part of apache#30248, using the `get_async_conn()` method as a context manager was removed in the AzureDataFactoryHook. The Python 3.7 tests all passed, but 3.8+ tests on `main` were failing. There is no reason to mock the `__aenter__` property since method is no longer used as a context manager.
josh-fell added a commit that referenced this pull request Mar 29, 2023
As part of #30248, using the `get_async_conn()` method as a context manager was removed in the AzureDataFactoryHook. The Python 3.7 tests all passed, but 3.8+ tests on `main` were failing. There is no reason to mock the `__aenter__` property since method is no longer used as a context manager.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants