TinyPilot's USB not recognized on macOS

[db39]

Description

Disconnecting and then reconnecting TinyPilot to a macOS-based machine via its USB cable causes TinyPilot to stop being recognized as a USB device. To restore functionality, you must reboot your TinyPilot. Alternatively, you can run sudo /opt/tinypilot-privileged/remove-usb-gadget and then sudo /opt/tinypilot-privileged/init-usb-gadget.

This behavior may only be present on Apple Silicon-based Macs. I could not reproduce this behavior on an Intel-based Mac running OS X 10.14 (very old).

This behavior was originally reported on a 2021 MacBook Pro M1 Pro running 14.6.1. I reproduced this behavior on a MacBook Air M2 running 14.4.1.

This behavior doesn't change when adjusting the Mac's USB accessory behavior. i.e., changing "Allow accessories to connect" to "Always", "Ask Every Time", "Ask for New Accessories", or "Automatically When Unlocked" doesn't affect the behavior.

There's a discussion thread on Apple's support forums related to USB hubs not working on macOS 14.4. So it may instead be an OS issue.

There are also similar Reddit threads reporting that some USB hubs no longer work correctly. Some users suggest that version 14.4.1 resolves these USB issues, but my machine on 14.4.1, and the TinyPilot forum user's machine on 14.6.1, both have this issue.

I haven't yet tried updating to Sequoia (15.0) - the user on the TinyPilot forum suggests the latest version of macOS also has this behavior.

What's the behavior that you expect?

I'd expect TinyPilot to successfully connect to the Mac's USB and correctly send mouse and keyboard input whenever you connect the USB cable.

What's happening instead?

When reconnecting TinyPilot's USB cable, the Mac no longer recognizes it as a USB device, causing a state where you cannot use TinyPilot's mouse or keyboard without rebooting TinyPilot.

What are the steps to reproduce this behavior?

1. Connect your TinyPilot to a Mac (possibly only Apple Silicon-based or running macOS >= 14.4)
2. Move TinyPilot's mouse or use the keyboard to show it's working
3. Disconnect the USB cable from the Mac
4. Reconnect the USB cable to the Mac
5. Attempt to move TinyPilot's mouse or type characters on TinyPilot's keyboard
   a. Nothing happens

Screenshots

The Mac's USB System Information after connecting TinyPilot initially:

The Mac's USB System Information after disconnecting and reconnecting TinyPilot's USB cable (this is identical to not having a connected USB device):

@jotaen4tinypilot & @jdeanwallace - Do you have any suggestions for additional tests I can perform here?

[jotaen4tinypilot]

This is pretty weird: I am able to reproduce this issue – however, only with a USB-C<>USB-C cable. If I’m using a USB-C<>USB-A cable along with a USB-A<>USB-C adaptor, then replugging the cable works just fine. I.e.:

• TinyPilot device—USB-C<>USB-C cable—Target Machine (M2 / macOS 14.6.1)
  • I can reproduce the issue, so the USB connection doesn’t recover after replugging the data cable.
  • 
• TinyPilot device—USB-C<>USB-A cable—USB-A<>USB-C adaptor—Target Machine (M2 / macOS 14.6.1)
  • I cannot reproduce the issue, so the USB connection recovers immediately after replugging the data cable, as expected.
  • 

I have no immediate idea what might be going on here, but my hunch would be that it could potentially be some low-levelled hardware/interface issue related to the involved USB versions/standards?

To assess the severity of this issue, it might also be worth to check whether this is an issue specific to Apple/macOS, or whether this also happens on other target machine kinds (i.e., operating systems and hardware vendors). Unfortunately, though, I don’t have any other non-macOS target machines at hand with USB-C ports to verify this.

[db39]

@jotaen4tinypilot - thanks for those tests!

I tried to reproduce your USB-C/USB-A behavior, and I can get it work, but only with specific steps.

If I disconnect and reconnect the USB cable on the TinyPilot side (Data port), the mac recognizes TinyPilot again.

If I disconnect and reconnect the USB cable while leaving the USB-A -> USB-C adapter connected to the mac, the mac recognizes TinyPilot again. Maybe this is because the USB-C connection never gets completely severed?

If I disconnect and reconnect the USB-A -> USB-C adapter (still connected to the USB-A side of the cable) from the mac, the mac doesn't recognize TinyPilot when reconnecting.

Using USB-C/USB-C directly doesn't work regardless of which side I disconnect/reconnect.

The only machine I have with working USB-C ports is my Mac, so I can't test USB-C/USB-C on a different machine, unfortunately. I may be able to try and get a USB-C port working on my testbench PC, but I may have to open up the machine and diagnose the issue, so it might not be worth it time-wise if someone else has access to one.

It sounds like a USB-A -> USB-C adapter is a usable workaround for now.

I think my next tests would be to either test USB-C -> USB-C on a non-mac machine, or try and find the underlying issue by looking at the Mac's logs when connecting / disconnecting (something like log stream but filtering USB?).

[jotaen4tinypilot]

I think it would be good if we’d manage to verify whether or not the pure USB-C<>USB-C connection is also an issue on non-macOS machines. My worry would be that the issue only surfaced in the context of macOS so far, just because USB-C ports happen to be more common on Apple devices. If it would turn out to be an issue strictly related to the cable and not strictly related to macOS, then this might change the priority of this issue for us. @jdeanwallace / @shalver-tp would you be able to help out here? (I.e., testing the reconnection behaviour on a non-Apple/non-macOS target device that provides a native USB-C port?)

Regardless of that, I think investigating the internal system logs sounds like a good idea, maybe that provides helpful clues.

In case we don’t manage to fix the issue, another workaround that we could consider providing would be some sort of shortcut/functionality for users to re-initialize the USB gadget via the web UI.

[scott-tp]

@jotaen4tinypilot

I just tested this setup:

TinyPilot Voyager 2a running 2.6.4

Plugged via USB-C <-> USB-C into an ASUS Laptop running Windows 10

I disconnected the USB-C from the ASUS multiple times, I always got control back immediately and heard the "ding" of a new USB device being connected.

[scott-tp]

@jotaen4tinypilot @db39 I also just ran this test:

TinyPilot Voyager 2a running 2.6.4

Plugged via USB-C <-> USB-C into an M3 Macbook Air running MacOS 14.5

I disconnected the USB-C from the Macbook multiple times and I always got control back immediately when reconnecting. I also tried disconnecting from the TinyPilot side, the behavior was the same - I got control back immediately.

When I first plugged it in, I was prompted to give control/access to this USB accessory.

[db39]

Thanks for testing all that, @shalver-tp!

This is pretty odd behavior. It seems like it's specific to certain Macs or OS versions. It doesn't seem like this issue is widespread since we've only had one report from TinyPilot users.

I wonder if different USB-C cables would make a difference?

I'll take a deeper look at how my Mac behaves when connecting / disconnecting TinyPilot and see if there's any information I can find that would point to an underlying issue an if there's anything we can do on the TinyPilot side to fix this.

[scott-tp]

As a note, the USB-C cable I was using came with an SSD hard drive.

[jdeanwallace]

Sorry for my radio silence on this issue. I don't have an Apple Silicon-based Mac (only the old Intel-based Mac) and I don't have any target machines with a native USB-C port (only USB-A ports).

[cghague]

I'm not in a position to test this, but I've got a lot of experience with macOS and Mac hardware. I suspect this isn't a USB-C versus USB-A issue, but rather that there's something amiss between using a USB 3.0 Type C port, a Thunderbolt Type C port, and either of those ports running in USB 2.0 mode. That might explain why only some Macs are affected and why using hubs and adapter cables might avoid the issue.

[db39]

Thanks for that info, @cghague! It sounds plausible that different modes could be the issue. Do you have any suggestions for how to test this?

[cghague]

@db39 - The tests above suggest that connecting through a USB hub or adapter prevents the issue from occurring, and you mentioned that your older Mac is unaffected. Is it a model that doesn't support Thunderbolt? If so, it's plausible that Thunderbolt is the culprit, and using a USB hub or adapter is sufficient to force the Thunderbolt port to stay in USB mode. An easy way to test this would be to find a Mac that has both native USB-A ports and Thunderbolt ports. My Mac Minis have that configuration, but unfortunately, I can't get to them for testing at the moment. Do we know if anyone else has one we can test with?

@scott-tp - We recently discussed adding some soft restart options, such as the ability to relaunch uStreamer, to the menu. Adding a "Reconnect USB" menu item might also be a good idea for issues like this one. I'll leave it to you to decide if that's worth the development time!

[jotaen4tinypilot]

We recently discussed adding some soft restart options, such as the ability to relaunch uStreamer, to the menu.

Just one thought here, as I happened to see your comment: while such a “soft-restart” functionality would be helpful for certain troubleshooting scenarios, I think we should be careful in how we label or where we put this. My thinking is, if we e.g. provide this as menu item, then it would be relatively prominent – which might leave the (undesired) impression that the user can expect the system to be somewhat unreliable by design, or which might lead to users potentially overusing this whenever something doesn’t work right. So we could maybe also consider making this functionality a little more hidden and discreet.

I don’t want to side-track this issue, though, so if such a soft-restart functionality is something we’d be interested in, we could maybe continue the discussion in a separate ticket.

[scott-tp]

Thanks for the discussion, all!

As far as "reconnect USB" and "Soft restart" - I feel like we can avoid adding those to the UI. However, this might be a great opportunity to leverage and highlight the upcoming Scripts Menu functionality.

[db39]

you mentioned that your older Mac is unaffected. Is it a model that doesn't support Thunderbolt?

The old one was a 2012 MBP which has Thunderbolt. Interestingly, Scott's new M3 Macbook Air doesn't seem to have this issue.

An easy way to test this would be to find a Mac that has both native USB-A ports and Thunderbolt ports. My Mac Minis have that configuration, but unfortunately, I can't get to them for testing at the moment.

Would you be happy to try this when you get access to your test setup? In the meantime, I'll take another look into this / do more testing I get chance.

[db39]

I did a couple of tests while collecting some logs to see if we could spot anything going on, but it doesn't look like they revealed anything.

I connected, disconnected, and then re-connected the USB cable from my TinyPilot to my Macbook Air. In test 1, I did this with a USB-C cable, and in test 2, I did this with a USB-C to USB-A cable and a USB-A to USB-C adapter (the workaround solution).

During the test, I was running log stream --predicate 'eventMessage contains "USB"'

Test 1 logs

Test 2 logs

The second test shows that TinyPilot reconnects correctly, but doesn't in test 1. Unfortunately, I can't see anything that suggests why.

I ran the tests again and couldn't find any Thunderbolt logs or anything like that.