Audio over AOA Issues

[khburdette]

Bug Report

With the new router service logic, media apps are connecting over AOA (USB) and unable to play media.

When starting the AOA protocol, the head unit will send a product ID (see AOA spec) which can be used to determine if the head unit supports audio or not. This information should be used in determining whether or not to connect a media app to a router service using AOA.

In our testing some devices don't support media over AOA. When connected over USB the device will still sent the audio via A2DP. Samsung devices were among the devices that observed this behavior.

Because of the inconsistency with devices, we decided not to support media over AOA on SYNC head units. I propose we provide a hot fix to not allow media apps to connect to a router service that's connected via AOA until we can better determine if both the head unit and device support media over AOA before connecting.

Worth noting that with the 4.8 library, we are able to specify transport(s) to be used. Older libraries using the router service will still connect via AOA when any router service connects via AOA. Another possible hot fix is to require apps using the 4.8 library to specify their transport and only allowing apps with 4.8+ to connect to a router service over AOA. Again, another temporary solution until we can determine if both the head unit and device support media over AOA and then connecting.

Reproduction Steps

1. Install app that supports AOA (Hello SDL)
2. Install media app using the router service
3. Connect phone via USB and select the AOA app when prompted on device

Expected Behavior

Media apps will only connect over AOA if both the device and head unit support it

Observed Behavior

Media app connects over AOA and is unable to play media

OS & Version Information

• Android Version: 9
• SDL Android Version: 4.8
• Testing Against: SYNC Gen 3

[joeygrover]

This is a rather complicated issue, so I apologize for the novel below.

First and foremost, the module implementing Core should deny registration to a MEDIA app if the media component isn't ready or available.

However, with the Gen3 system it seems that is not the case. While it is a much more intended action, a MEDIA app could have used the pre-multiplexed AOA connection and still registered with the module and hit the same issue.

Moving onto the library side, the router service is merely a pipe for SDL messages to be sent to and from SDL apps. It has the bare minimum knowledge of the apps that are binding to it by design. This means it has no knowledge of which apps have certain requirements as in this case a method for audio arbitration. It will send the connected intent and allow usage to any type of app , it will be up to the app (or client side of the binding relationship) to decide if the current situation is acceptable to use. This can be seen where apps have the ability to require a high bandwidth transport.

Neither of the two presented solutions would be acceptable. There is no specific requirement that an IVI system has to support audio over AOA in order to use AOA for SDL. An IVI system could use AOA for SDL and send audio through the AOA, bluetooth A2DP, or even an aux cord. While the latter is not likely, the second is a very real scenario. Not allowing MEDIA apps to connect over AOA entirely would block this type of integration. Which would make the solution unacceptable to any adopter/SDLC member that has implemented this type of connection.

Furthermore, one issue with SDL is discoverability. If we prevent apps from connecting in any scenario there is less of chance that app will be discovered. Providing a message prompt to the user to connect an audio source in this instance would be a better end goal in my opinion.

Based on this information I think we have to create a better solution. One potential work around is giving apps the ability to select if they requireAudioConnection into their multiplexing transport config. The library itself can check the Android AudioManager for connected devices and ensure that at least one of the supported audio transports are connected. However, finding the correct timing for this might be a little tricky as we don't have any requirement, for example, saying A2DP has to be connected before SPP. If we are early in the process we may miss a potential acceptable scenario which is also a concern.

Alternatively to AudioManager, the Android MediaRouter API is the more direct path to find where audio is being router, but it was introduced in api level 16 (Android 4.1) so it wouldn't cover all Android devices. According to Android's distribution dashboard (which oddly hasn't been updated since October 26, 2018) this would cover 99.5% of Android phones though.

I would recommend taking options to the SDLC steering committee next week and explaining the situation. Then give the members a chance to investigate and review this issue, and vote on a solution the following meeting.

1. Do Nothing. Core module should reject MEDIA app registration if no audio arbitration method is available.
2. Tell Media apps to remove USB as an acceptable connection from MultiplexTransportConfig
3. Library does not allow Media Apps to use USB at all from client side of router service binding relationship
4. Increase the amount of data the router service collects to prevent MEDIA apps from connecting over USB
5. Add additional flag, requireAudioConnection, to MultiplexTransportConfig and library checks AudioManager if an acceptable audio output device is present (would need SDLC to define which devices are acceptable See Andorid's AudioDeviceInfo)
6. Same as 5. but with MediaRouter
7. Similar to 5, add additional option to config to send Show message to inform developer to connect audio source vs just closing their service if no acceptable audio device is present.
8. Same as 7 but with MediaRouter

[khburdette]

I think options 4-8 are all good options for a more permanent solution and can be discussed in the SDLC steering committee. I also think we need to come up with a solution for media apps using the router service on library versions <4.8. Can we brainstorm a hotfix solution for this?

[theresalech]

As discussed during the 2019-05-07 Steering Committee call, members will review this issue and possible solutions provided by the Project Maintainer. Members should comment with additional solutions if applicable and plan to vote on a solution during the 2019-05-14 Steering Committee meeting. Members should also advise if they are using AOA and if they rely on other sources of audio routing outside of AOA, and take this into consideration when determining the best solution. We will plan to include a fix for this issue in the 4.9 Release, as agreed to during the 2019-05-07 Steering Committee call.

[ashwink11]

As suggested in solution 5, I agree that library should ensure that at least one of the supported audio output device is connected. However, instead of using flag requireAudioConnection, can we use AppHMIType::MEDIA?

In case we keep this flag, what would be the behavior if requireAudioConnection is not set, but App is an MEDIA App and still tries to route Audio through IVI?

In addition to these solutions mentioned in 4-8, I think, we should also consider solution 1 as well. Core module should reject MEDIA app registration if no audio arbitration method is available.
This would be preventive step to make sure that IVI does not go in undefined state, if App does not select requireAudioConnection , but registers on IVI as an MEDIA App and try to route Audio.

[khburdette]

An example of how AOA is currently behaving:
Install (from the play store): App Catalog and Sygic Car Navigation
Install Hello SDL with 4.8.1 library version

Scenario 1:

• Plug in device
• Select Hello SDL on the AOA popup:
  • observed behavior: App Catalog and Hello SDL show up on the head unit

Scenario 2:

• Plug in device
• Select Sygic on the AOA popup:
  • observed behavior: Sygic shows up, Hello SDL has foreground service but never shows up
  • when you unplug the device, Hello SDL never removed it's foreground notification, so this is putting Hello SDL into a bad state

This might weigh into our possible AOA hotfix, because I don't think this is ideal ...

[joeygrover]

I hesitate to force a developers integration, so simply saying that MEDIA apps can't connect over AOA when audio isn't present is a hard rule and prevents some discoverability. Instead, we could say the requireAudioConnection will be set to true by default for MEDIA apps, but can be overridden if set to false by the developer.

I'm 50/50 on performing 1 along with any from 4-8. It creates two solutions to the same problem, but 1 is more straightforward. I'd leave it to the SDLC to decide.

As stated before though, a MEDIA app could put up an alert of information on screen that tells the user to connect a media pipe of somesort.

@khburdette I don't think that has anything to do with the specific issue called out. I would get logs to see why that's happening first as it would appear to be a separate problem. What happens when you change the Hello SDL hmi type to NONMEDIA? Same behavior? I would open another issue to further discuss what you're seeing.

[khburdette]

Yes, this also happens with NONMEDIA. I agree this needs to be another issue. I'll have to grab some logs

[theresalech]

An SDLC member reached out via email to confirm that they are comfortable with option 1, and 4-8, as they have assumed a Bluetooth A2DP connection was a prerequisite for SDL apps to register.

[khburdette]

I just tested with a Samsung Galaxy S9 running Android 8.0 and it doesn't support audio over AOA. Instead it sends the audio to A2DP.

My Pixel 3 XL running Android 9 does send the audio over AOA.

@theresalech, even if A2DP is a prerequisite, it needs to be very clear that different devices function differently, so having a media app connected via AOA will have different behavior depending on the device. I'm not sure how an OEM can figure out where the device is sending the audio to be able to actually play it.

[joeygrover]

The IVI system will have to support the Audio over AOA though for audio to get routed by the phone through AOA though right? So the IVI system would know it supports AOA audio and would follow the a priority (if AOA audio, use that. Else if A2DP use that). Or it only supports one and uses that one. The app developer doesn't specify where the audio is being streamed as it is handled by the OS (AFAIK).

I for sure don't believe A2DP should be a prerequisite, but I agree it does bring up a need for clarification around audio arbitration prioritization.

[khburdette]

I dug of some of our findings when we were looking to introduce audio over AOA for SYNC3:

Phone paired via bluetooth and connected via AOA:

• Samsung devices route audio through A2DP
• Pixel devices route audio through AOA

Phone NOT paired via bluetooth and connected via AOA:

• Samsung devices route audio through AOA
• Pixel devices route audio through AOA

So the issue is that where the audio is being streamed is handled by the OS, but different handsets have different behavior.

If the head unit only supports AOAv1, the devices do route the audio through A2DP, which seems to be the only thing consistent here.

[khburdette]

Can we rev the router service version and not allow older router services to connect via AOA? This way only AOA apps would be required to upgrade.

This would also mean that when other apps finally upgrade they can decide to use bluetooth only or benefit from one of the more permanent solutions listed above.

[joeygrover]

I'm not following your comment. You want to remove the AOA connecting ability for the current version of the router service and only allow it for new versions of the router service? If that's what you mean, that won't work. All apps would still have to upgrade to have the functionality removed from that version of the router service, as well as that's a breaking change. And even still, if an AOA app is installed with the newer router service that allows the AOA connection, all the other apps will still bind to it and we would be back in this situation.

[KainGC]

Hi @joeygrover,

Can you please summarize the latest idea and conversation regarding this issue here in the ticket?
I know that some detail was included in the latest meeting minutes but anyone looking just at this ticket does not know the current status and it could cause some confusion.

[joeygrover]

The issue as described between the first two comments still remains true including the specific solutions.

The steering committee is trying to understand a few things:

1. Does anyone actually use AOA multiplexing for apps?
2. Do any members use Media apps over AOA?
3. Do any members implement audio over AOA or always rely on a different audio path, eg A2DP BT?

There is a PR attached to this issue as can be seen on this page. It currently implements solution 5
with some modifications as Android had unrealized limitations. There is ongoing work to attempt also bolting on number 4 but using the same logic as number 5 to only prevent Media apps from connecting when there is no audio streaming device attached.

If we don't get feedback, the easiest route will to be removing MEDIA apps from connecting over AOA regardless of any other audio streaming ability. So it's imperative that members speak up on their integrations and give feedback.

Is there anything else you'd like to see posted?

[joeygrover]

It should be noted that this issue will need to be voted on with clear actions by tomorrow's meeting (6/11) to be included in the next release of this library. If a decision can't be reached then it will have to be included in a later release.

I would recommend reviewing #1085 for its merits and determining if this is an acceptable path forward. The biggest drawback to this approach is that while it can prevent MEDIA apps from binding to a router service if no Audio output device is attached and unregister if that device is detached, it can't monitor for when an audio device attached after the router service sends out the SDL Enabled intent. Therefore, apps need to be manually opened on the phone to bind to the RS. However, developers could set up their own broadcast receivers to know when devices attached and make a call to the SdlBroadcastReceiver.queryForConnectedService(context).

I have continued to investigate ways to make the router service handle a similar functionality but it leads to very unstable behavior as well as a good amount of extra code and state tracking as this was not something the service was designed to do.

[jthrun]

@joeygrover,
I have a couple questions.

We have at least one app that is both a media app and a navigation app. Will this type of dual-purpose app also be unregistered/not bind to an AOA-only connected router?

What happens when a router has both a BT and an AOA connection? I think media apps will connect normally but just want confirmation.

[khburdette]

@joeygrover I think this is an acceptable path forward

[joeygrover]

@jthrun the default behavior implemented in the PR is that if an app is found to be a MEDIA app, the flag will be set to true that it requires an audio output device. If there isn't one attached during the time of a transport connection, the app will close its service down. This flag can be set by the developer though and override that behavior. So the case you mentioned, since you can accept the case where no audio is required you would set the flag like this multiplexTransportConfig.setRequiresAudioSupport(false);. If the flag is set in the MultiplexTransportConfig, the library will take that value rather than attempting to assign a default one. Any other app type will be set to false by default.

This PR actually doesn't check AOA vs BT, it only checks that there is audio support. So if any of the transports are connected, the dependency remains the same as what the flag is set to either by default of the developer.

@khburdette thank you for looking into it. I do realize this is not the best case scenario in terms of easiest fixes for the router service, but I do believe that the instability presented by attempting to put this logic in the RS is not worth it. The PR is a much cleaner solution and doesn't impact RS stability.

[BrettyWhite]

Closed with #1085