Unclosed aiohttp ClientSession and TCPConnector after AzureDataFactoryRunPipelineOperator execution

[stolarekms]

Apache Airflow Provider(s)

microsoft-azure

Versions of Apache Airflow Providers

=10.4.0

Apache Airflow version

2.10.2

Operating System

Official Airflow image apache/airflow:2.10.2-python3.12 hosted on AKS

Deployment

Official Apache Airflow Helm Chart

Deployment details

Official Airflow image apache/airflow:2.10.2-python3.12 hosted on AKS

What happened

This issue resembles Unclosed connector / client session when using asyncio #13242
However, it occurs when triggering an Azure Data Factory pipeline via Airflow using the AzureDataFactoryRunPipelineOperator.

Regardless of whether the pipeline completes successfully or fails, the following error is consistently logged:

[2025-09-16, 15:08:11 CEST] {base_events.py:1820} ERROR - Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f80f1a34380>
[2025-09-16, 15:08:11 CEST] {base_events.py:1820} ERROR - Unclosed connector
connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x7f80f1bb1970>, 102727.274540543)]']
connector: <aiohttp.connector.TCPConnector object at 0x7f80f1a34b00>

What you think should happen instead

Displaying the error message is not intended in this context.

How to reproduce

1. Use AzureDataFactoryRunPipelineOperator to trigger an ADF pipeline.
2. Wait for the pipeline to either succeed or fail.
3. Observe the logs — the error appears regardless of the pipeline outcome.

Anything else

This occurs consistently upon task completion, irrespective of the pipeline's success or failure.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[yash1thsa]

Hey @jroachgolf84 . Would love to take this up if its open for newbies.

[jroachgolf84]

@yash1thsa - please feel free to open up a PR for this issue!

[yash1thsa]

@jroachgolf84/@stolarekms i tried replicating this in local/azure with the given version, I dont see any error msgs. Any pointers to reproduce this ll help

[stolarekms]

@yash1thsa
Thanks for checking! I was running this using the AzureDataFactoryRunPipelineOperator in deferrable mode - maybe that’s the key to reproducing the issue.

[yash1thsa]

@stolarekms Thanks for the information. I was able to replicate the issue. I also did some digging at the code level. In azure provider , the aiohttp client is not directly called in the trigger code. Instead it is initiated via azure-mgmt-datafactory SDK. Should we point to a newer version? Any direction would help . - azure-mgmt-datafactory>=2.0.0

[stolarekms]

@yash1thsa
Thanks for looking into this and confirming the root cause.
While updating the azure-mgmt-datafactory SDK to >=2.0.0 might help in some cases, I believe it would be more robust to address this issue directly within the apache/airflow provider codebase. There are a few reasons for this:

• SDK upgrade might not be sufficient – the newer version may still not fully resolve the issue or might not yet be implemented in the provider logic.
• Potential version conflicts – bumping the SDK version could introduce compatibility issues with other packages in the Airflow ecosystem.
• Long-term maintainability – fixing it at the provider level ensures consistent behavior regardless of the SDK version used.

What do you think about this approach?

[yash1thsa]

@stolarekms thanks for the response. Let me dig up some more. Will get back to you with some code to discuss further

[arkadiuszbach]

That's the same problem as in: https://github.com/apache/airflow/pull/52119/files

Azure SDK is exposing close, but its not called, it has to be called once we are done with the client or "with" context manager used.

Client creation:

airflow/providers/microsoft/azure/src/airflow/providers/microsoft/azure/hooks/data_factory.py

Line 1155 in 28e49e7

self._async_conn = AsyncDataFactoryManagementClient(

close not called:

• airflow/providers/microsoft/azure/src/airflow/providers/microsoft/azure/hooks/data_factory.py

  Line 1216 in 28e49e7

  client = await self.get_async_conn()

• airflow/providers/microsoft/azure/src/airflow/providers/microsoft/azure/hooks/data_factory.py

  Line 1182 in 28e49e7

  client = await self.get_async_conn()

• airflow/providers/microsoft/azure/src/airflow/providers/microsoft/azure/hooks/data_factory.py

  Line 1162 in 28e49e7

  async def refresh_conn(self) -> AsyncDataFactoryManagementClient: # type: ignore[override]