Error:asyncio:Unclosed connector

[andersbogsnes]

What happened:
When running upload/download scripts using adlfs, we get a warning on every run:

ERROR:asyncio:Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f2012e55820>
ERROR:asyncio:Unclosed connector
connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x7f2012ebcfa0>, 10699.800393553)]']
connector: <aiohttp.connector.TCPConnector object at 0x7f2012e5585

While it is just a warning, it can be scary for end-users as it seems that something went wrong.

What you expected to happen:
No warning should be raised

Minimal Complete Verifiable Example:

import fsspec
from adlfs import AzureBlobFileSystem

fs: AzureBlobFileSystem = fsspec.filesystem("az", account_name="andersdatalake",
                                          account_key="my_account_key")

with open("hello_fsspec.txt", mode="w") as f:
    f.write("Hello from fsspec!\n")

fs.put("hello_fsspec.txt", "/raw/hello_fsspec.txt")

Anything else we need to know?:

My immediate guess is that it comes from not closing the credentials transport session per this issue: Azure/azure-sdk-for-python/issues/8990 and this PR: Azure/azure-sdk-for-python/pull/9090

If that is the source, I imagine the fix would be to call close on the credentials object, but I'm not sure where in the lifecycle to do that.

Environment:

• adlfs: 0.5.5
• Dask version: N/A
• Python version: 3.8
• Operating System:Ubuntu 20.04
• Install method (conda, pip, source): pip

[Timost]

I've just hit this issue too, I think this is because Azure async clients should be used as context managers or their close method should be called properly. It seems adlfs does not do this properly.
Azure/azure-sdk-for-python#13242

[Darune]

I'm not sure i have the best solution, i checked what was doing gcsfs regarding this.
Read the documentation about weakref.finalize() this seems to be the way to close aio clients that connects in constructors.

[trymzet]

This is still broken in 0.7.5:

fs = AzureBlobFileSystem(
    account_name=gen2_storage_account_name,
    tenant_id=tenant_id,
    client_id=gen2_client_id,
    client_secret=gen2_client_secret
)
fs.ls("")

Produces:

['my_container']
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7fb45eef6a60>
Unclosed connector
connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x7fb45eee8fa0>, 3901736.601287218)]']
connector: <aiohttp.connector.TCPConnector object at 0x7fb45eef69d0>

[aaravind100]

This is still broken in 0.7.5:

fs = AzureBlobFileSystem(
    account_name=gen2_storage_account_name,
    tenant_id=tenant_id,
    client_id=gen2_client_id,
    client_secret=gen2_client_secret
)
fs.ls("")

Produces:

['my_container']
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7fb45eef6a60>
Unclosed connector
connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x7fb45eee8fa0>, 3901736.601287218)]']
connector: <aiohttp.connector.TCPConnector object at 0x7fb45eef69d0>

This still happens in 2022.2.0. The workaround I found is to explicitly close the fs.credential client at the end of the program.

import asyncio

fs = AzureBlobFileSystem(
    account_name=gen2_storage_account_name,
    tenant_id=tenant_id,
    client_id=gen2_client_id,
    client_secret=gen2_client_secret
)
fs.ls("")

asyncio.run(fs.credential.close())  # <-- this