Connecting to device which is already connected fails (Device with address ... was not found)

[eudoxos]

• bleak version: git develop branch
• Python version: 3.8
• Operating System: Ubuntu 20.04
• BlueZ version (bluetoothctl -v) in case of Linux: 5.53

Description

I am first-time bleak user and want to talk to a GATT server with known address. Connection can be established when the device is currently unconnected (the tray shows . When the device is connected already (e.g. from previous run), connection fails with bleak.exc.BleakError: Device with address F9:10:AE:E7:9E:40 was not found.

What I Did

I tried to only call BleakClient conditionally when BleakClient.is_connected returns True. But the device is reported as unconnected.

I run subprocess.call(['bluetoothctl','disconnect',addr]) before calling BleakClient.connect() (returns quickly when not connected) but that somehow does not seem right at all.

What is the best practice for this scenario?

This is a MWE:

import bleak, asyncio, subprocess
addr='F9:10:AE:E7:9E:40'
# without this, already connected device won't connect
if 1: subprocess.call(['bluetoothctl','disconnect',addr])

async def awrap():
    async with bleak.BleakClient(addr) as cl:
        print('Connection established')
asyncio.run(awrap())

and this is the error output (without the subprocess call):

Traceback (most recent call last):
  File "min2.py", line 9, in <module>
    asyncio.run(awrap())
  File "/usr/lib/python3.8/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "min2.py", line 7, in awrap
    async with bleak.BleakClient(addr) as cl:
  File "/home/eudoxos/.local/lib/python3.8/site-packages/bleak/backends/client.py", line 59, in __aenter__
    await self.connect()
  File "/home/eudoxos/.local/lib/python3.8/site-packages/bleak/backends/bluezdbus/client.py", line 98, in connect
    raise BleakError(
bleak.exc.BleakError: Device with address F9:10:AE:E7:9E:40 was not found.

[eudoxos]

Can you give a full code example to reproduce the problem?

Just added as edit.

[hbldh]

If the device is already connected, then it will not advertise and can therefore not be found by Bleak. You will never be able to connect to an already connected device.

If you e.g. press Ctrl-C in your program, you might be left with a connected device. Try to ensure a clean exit.

[eudoxos]

Ok, thanks for explanation.

Is disconnecting some/all connected devices (which I now achieve via bluetoothctl disconnect) beyond Bleak's scope? I might use BlueZ over DBus directly for that.

[dlech]

Duplicate of #315?

[eudoxos]

Duplicate of #315?

That one is about to prevent stale connection left behind.

What I am after is establishing connection if there is stale connection hanging around already, for whatever reason.

Or at least a better error message than "Device not found" would be nice ("Device already connected" would be much more helpful.)

[hbldh]

The problem is seems to be exactly that: that your application leaves stale connections after it exits. Bleak only handles connections it has created by itself, it is not a Bluetooth Manager in that sense. Thus, it cannot say "Device already connected", because it doesn't know about such things.

But if you use the async with bleak.BleakClient(...) solution, you should not experience that. Unless you e.g. exit your program with Ctrl-C.

[eudoxos]

The problem is seems to be exactly that: that your application leaves stale connections after it exits.
But if you use the async with bleak.BleakClient(...) solution, you should not experience that.

The async context manager does not work for me as I inherit from BleakClient and need to keep the instance connected while other parts of the code run. I define destructor like def __del__(self): asyncio.run(self.disconnect()), which eliminated most stale connections, but not all. Like if the app is killed or someone connects e.g. through the Bluetooth applet.

Anyway, I see what you mean. I thought Bleak would ask BlueZ over DBus whether the connection exists already but I am okay to do it myself.

Thanks for the advice and helpful hints, let me close this now. And thanks for Bleak :)

[newAM]

Sorry to comment on an old issue, but I am looking for a similar solution for newAM/idasen#29

I thought Bleak would ask BlueZ over DBus whether the connection exists already but I am okay to do it myself.

Is there some code for this that you could point me to?

[hbldh]

Not that I am aware of. It is still outside the scope of Bleak.

[eudoxos]

Hi @newAM, I am using this to list already connected devices (for BlueZ):

def bluez_connected():
    import dbus
    bus=dbus.SystemBus()
    manager=dbus.Interface(bus.get_object('org.bluez','/'),'org.freedesktop.DBus.ObjectManager')
    objs=manager.GetManagedObjects()
    return [str(props['org.bluez.Device1']['Address']) for path,props in objs.items() if 'org.bluez.Device1' in props and props['org.bluez.Device1']['Connected']]

If the device I want to connect to is connected already, then I disconnect like this (could be also done over DBus, this is a quick hack):

try:
    subprocess.call(['bluetoothctl','disconnect',addr,timeout=2)
except (subprocess.CalledProcessError,subprocess,TimeoutExpired):
    # do what you need to do

Afterwards, Bleak will connect happily.

[dlech]

I've done a bit of digging and it does look like it is possible to enumerate already connected devices on all OSes.

• BlueZ already caches all devices, so we can just use the Connected property as mentioned above.
• CoreBluetooth can retrieve a list of connected peripherals
• Windows can enumerate connected devices.

So perhaps we could add an async static method to BleakScanner that gets connected devices. This would return a list of BleDevice just like the discover() method. find_device_by_address() could then internally first check already connected devices, then star scanning if the device is not found.

[hbldh]

Good work! I feel reluctant to "adopt" devices that other BLE solutions have set up, e.g. there might be notifications set up elsewhere that are left if Bleak disconnects. Not sure if that even is a problem, though.

If it is possible to do, why not. Personally I am not sure the actual value added is worth it, but the fact that I won't use it is not good reason to avoid it.

[leeprevost]

I think I"m running into this same problem or set of symptoms. I'm using the OpenGoPro tutorials and their connect_ble.py routine that uses bleak under the covers. I"m also on Win10.

link to github folder

I can use this routine every time to discover and pair a device that is in pairing mode. But, after it goes to sleep and I restart the gopro, I routinely get the device not found error even right after it was discovered by the BleakScanner but afterwards when the client attempts to connect, I get the device not found error.

INFO:root:Scanning for bluetooth devices...
INFO:root: Discovered:
INFO:root: Discovered: GoPro 2659
INFO:root: Discovered: 9067640A2A0A33ADE9
INFO:root:Found 1 matching devices.
INFO:root:Establishing BLE connection to GoPro 2659...
Traceback (most recent call last):
File "", line 1, in
File "C:\Program Files\JetBrains\PyCharm 2020.3.3\plugins\python\helpers\pydev_pydev_bundle\pydev_umd.py", line 197, in runfile
pydev_imports.execfile(filename, global_vars, local_vars) # execute the script
File "C:\Program Files\JetBrains\PyCharm 2020.3.3\plugins\python\helpers\pydev_pydev_imps_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "C:/Users/lee/Dropbox/python sandbox/dredge_pro/gopro/connect_ble.py", line 125, in
asyncio.run(main(args.identifier))
File "C:\Users\lee\Anaconda3\envs\gopro\lib\asyncio\runners.py", line 44, in run
return loop.run_until_complete(main)
File "C:\Users\lee\Anaconda3\envs\gopro\lib\asyncio\base_events.py", line 616, in run_until_complete
return future.result()
File "C:/Users/lee/Dropbox/python sandbox/dredge_pro/gopro/connect_ble.py", line 119, in main
client = await connect_ble(dummy_notification_handler, identifier)
File "C:/Users/lee/Dropbox/python sandbox/dredge_pro/gopro/connect_ble.py", line 90, in connect_ble
await client.connect(timeout=30)
File "C:\Users\lee\Anaconda3\envs\gopro\lib\site-packages\bleak\backends\dotnet\client.py", line 160, in connect
raise BleakError(
bleak.exc.BleakError: Device with address GoPro 2659 was not found.

[egnor]

Hello friends and much gratitude 🙏 for Bleak which seems to be far and away the most thoughtfully designed BT/LE interface layer for Python!

There is quite a bit of discussion about the "disconnected device issue" around:

• Device gone forever after Ctrl-C or kill #315
• Enumerating Connected Devices #412
• Bleak not disconnecting automatically when run and stopped/interrupted with PyCharm #613

And in other projects:

• Failed to connect when device is already connected newAM/idasen#29
• Error on running gopro-video demo gopro/OpenGoPro#14
• Bleak async loops must be exited gracefully in order to prevent connection problems gopro/OpenGoPro#39

This seems to be the most promising thread (including some great positive suggestions by @dlech above). I'm happy to help if there's a good agreeable direction! I apologize profusely if I'm just stirring the pot uselessly. Please check my understanding of the situation and the possible solutions, corrections much appreciated...

My understanding: there is an issue (quirk?) with the bluez (Linux) implementation (and possibly other platforms?), where if an app is connected to a device and then terminates without disconnection, the device remains connected and the connection is kept alive by bluez. That means the device no longer shows up in discovery, and becomes an "orphan" unavailable for use until the device is restarted, the computer is restarted, bluetoothd is restarted, or some bluetoothctl incantations are made.

(OPEN QUESTION: What does happen on other platforms currently?)

SOAPBOX RANT 📦 (skip this blockquote you don't want to argue the subject)

I believe that a system component which requires programs to perform "clean" shutdown to avoid broken system state (that cannot be recovered by an app restart) is extremely undesirable and seems like important to fix. "You can force-kill and restart apps without leaving the system broken" is a core design principle of all modern operating systems. When you kill an X11 (or Wayland) app, its window goes away; when you kill a socket using app, the socket is closed; when you kill an app talking to a USB device, the device is released; this is a widely observed principle.

I believe that careful clean shutdown with signal handlers and the like is not a complete solution, because program death is a fact of life and a design feature of the operating system. Programs seg fault, they run out of memory, they get killed by ornery users, they are force-killed by window managers, they lock up and need to be terminated, and so on. This is not a wildly exceptional circumstance, program termination happens all the time on every modern OS to every computer user, expert and novice alike. This is especially true of user-facing apps which are likely to be using a framework like Bleak. Asking every user to understand bluetoothctl incantations to "liberate" the device seems unreasonable, and more of a burden than almost any other device subsystem requires. (Serial port locks used to be like this, and it was a nightmare, and eventually got mostly fixed.)

At some level this is the application's responsibility, but a framework/library such as bleak needs to make it possible to make resilient and robust apps, otherwise it is a much less useful framework/library.

END SOAPBOX 🧼

Even if people agree with that rant, the question is how to deal with it. There seem to be a few options:

Cleanup hygiene: Try to ensure bleak-using apps all cleanly disconnect when exiting for any reason. While probably good practice, I believe this is insufficient (see rant above).

Enumerate connected devices: Add some cross-platform facility to enumerate connected Bluetooth devices, and also allow them to be force-disconnected. This is a little problematic because how exactly is an app supposed to know which devices are "orphaned" and up for "adoption", and which are legit in use by some other app? As a messy solution, bleak could stash a datum saying which devices it opened and the PID it had, so future processes know which devices are candidates for "reclaiming"... but that sounds complex and subject to its own race conditions and failure modes.

Arrange for auto-disconnect: In an ideal world, bluez (and similar system layers) would notice that the requesting app has gone off the bus, and disconnect any devices that app had requested connections to (at least for apps which request this service). However, that requires cooperation on bluez's part. I'm not well enough read into the semantics of dbus to know how big a deal this would be. (OPEN QUESTION: Is there any hope in this direction?)

Other: ???

Again, I really, really like Bleak and am enjoying using it and have a ton of appreciation for the people giving their free time to make a not-so-shiny area of the system shinier! I'm just trying to figure out how to untangle this particular knot so that I can make robust apps and services.

[dlech]

My understanding: there is an issue (quirk?) with the bluez (Linux) implementation (and possibly other platforms?),

There is an open issue at BlueZ for this: bluez/bluez#89. It sounds like there is agreement that it needs to be fixed but no one has stepped up to do the work.

On other OSes, if multiple apps or the OS itself was also using a BLE device, then it would be possible to have a device that didn't actually disconnect when the Bleak program exits (even if it exited "cleanly"). But I don't recall any issues raised with this behavior, so I'm guessing it is not common.

I don't have specific knowledge of the internals of Windows/Mac but I assume that when you connect a device, it creates an OS handle (like a file handle) for the connection. If the program crashes, the handle is automatically released by the OS. If this was the only handle to the Blueooth device, then the device is disconnected.

This also means that BlueZ is the only backend that could "force disconnect" a device as suggested. So since this isn't available cross-platform, probably it isn't the best choice for Bleak. Instead, I think it makes sense for Bleak to be able to enumerate and "connect" to already connected devices. This would allow working around the BlueZ issue (users could create their own force-disconnect if they think that is the best solution). But at the same time, the feature stands on its own (e.g. connecting to a BLE mouse that is already connected to the OS).

[egnor]

(As an analogy, scanning (discovery) seems to be reference counted based on active sessions in bluez.)

What might the connected-devices API look like? A classmethod like BleakScanner.connected_devices() which would return a list of BLEDevice (in the same format as discovered_devices) which could be passed to BleakClient to "pile on"? (And perhaps the BleakClient address-based constructor could check that before running discovery.)

As it stands, BleakScannerBlueZDBus does collect the relevant data (in self._cached_devices), and that collection process would presumably be shared under the hood with connected_devices()? Based on #367 (comment) it seems like this ought to be doable for other platforms as well?

There's some subtlety around the effect of client.connect() and client.disconnect() (and equivalently(?) async with) for such a "pile-on" client. If some backends have ref-counted connection and others have single-global-state connection, it could be tricky for apps to do the right thing. If nothing else we should make sure to carefully document the gotchas here.

[leeprevost]

with the bluez (Linux) implementation (and possibly other platforms?)

Definitely Windows too! See post above.

[dlech]

What might the connected-devices API look like?

Yes, something like this.

There's some subtlety around the effect of client.connect() and client.disconnect()

Indeed. For Mac/Windows, I think the existing code will just "do the right thing". For BlueZ, we will probably just have to pick a behavior that works for most users until it gets fixed upstream, i.e. if the device is already connected, then return from the Bleak connect method successfully without calling the D-Bus connect method (which would fail with "already connected") - and it should set a flag so that the disconnect method doesn't actually call the D-Bus disconnect method since another app may be using the device.

One thing that I'm not sure about though is how Mac/Windows would handle the BleakClient disconnect method when another app is still using the device. In all backends, Bleak currently waits for feedback that the device has actually been disconnected. Do Mac/Windows send artificial disconnected signals to say that our specific OS handle has been released? Or should Bleak not actually be waiting for this feedback? We will have to do some testing to find out.

[egnor]

Definitely Windows too! See post above.

It would be good to nail down what's happening there. Is there a way to check out the "underlying" system bluetooth state (as one does with bluetoothctl on Linux) to see what's going on? https://docs.microsoft.com/en-us/uwp/api/windows.devices.bluetooth.bluetoothledevice.frombluetoothaddressasync makes it sound like connection is a more... automatically managed... thing on Windows; I'm not sure how it relates to discovery.

[leeprevost]

Definitely Windows too! See post above.

It would be good to nail down what's happening there. Is there a way to check out the "underlying" system bluetooth state (as one does with bluetoothctl on Linux) to see what's going on? https://docs.microsoft.com/en-us/uwp/api/windows.devices.bluetooth.bluetoothledevice.frombluetoothaddressasync makes it sound like connection is a more... automatically managed... thing on Windows;II'm not sure how it relates to discovery.

I would be glad to lend a hand but I think @tcamise-gpsw is into this deeper while working on #gopro/OpenGoPro#39 and is probably in better position to shed light on the multi-platform issues.

[hypernewbie]

I vote that force disconnection is the right thing to add as an option here.

In most applications, such as sensors, the user with the active intention program is the one that the bluetooth device should be paired to. If user is waiting for BT sensors to connect by pressing the connect button, then in most apps the expected behaviour is that the user connects the app they're pressing the connect button on to the BT sensors because they pressed the connect button on the app they're looking at.

Only being able to connect to devices that don't already have a connection is not a pathway to building any sort of robust application in practice. Disconnections happen, things get paired to something else, and when user bring up the UI to pair again to press things like the "connect" button something is already not being paired right and the user wants to pair it to the current program there then and now, or else we wouldn't be calling the connect code, or the current program won't even be running in the first place.

Please see examples like Wahoo and Garmin phone apps as well as Zwift for examples of ship-ready robust bluetooth logic.
This is currently not that.

[buganini]

A device connected before bleak's lifecycle is a normal scenario, eg., a BLE keyboard with an extra configuration service, it will always be connected by the system automatically, and a configuration program that is only executed when needed.
Without the function to retrieve paired/connected devices, user will have to forget the device in this case.

[buganini]

https://github.com/buganini/bleak/tree/feature/CoreBluetooth_retrieveConnectedPeripheralsWithServices
I have a working version on macOS

[buganini]

On linux scanner just reports paired devices, but I have problem on services discovery, probably a bluez issue

[eudoxos]

This is BLE device listing under Windows with winrt https://stackoverflow.com/a/72045486 (we sponsored that code, and it is actually used in production, with some non-essential adaptations).

[pe224]

This is BLE device listing under Windows with winrt https://stackoverflow.com/a/72045486 (we sponsored that code, and it is actually used in production, with some non-essential adaptations).

Thanks, this is interesting. But am I correct that it only lists paired BLE devices? If a connection has been established (e.g. via bleak) without pairing, it does not appear for me in this listing.

[eudoxos]

If a connection has been established (e.g. via bleak) without pairing, it does not appear for me in this listing.

I might be forgetting (the primary platform is now Linux, Windows being used sporadically) but don't remember the necessity to pair the device before connecting; but it might be only for this industrial device.

[tswaehn]

Just wondering if there is news about it?