Implement microwakeword detection

[TimoPtr]

Summary

Inspired from ESPHome implementation and from the great work from @brownard in https://github.com/brownard/Ava.

I've rework the C++ part to not duplicate the source in our codebase and tweak the JNI to have a clearer separation of concern.

The Kotlin part has been generated with the help of Claude. For now it's only accessible through the DevPlaygound in debug.

We should not merge this in main until we have a proper plan on how to enable/disable this.

I've added a small instrumentation test that is going to be helpful verify that bumping NDK won't break the version we target.

I've enabled the compiler optimization also in debug since we want debug to behave the same as release in our case and the native code is small.

Checklist

• New or updated tests have been added to cover the changes following the testing guidelines.
• The code follows the project's code style and best_practices.
• The changes have been thoroughly tested, and edge cases have been considered.
• Changes are backward compatible whenever feasible. Any breaking changes are documented in the changelog for users and/or in the code for developers depending on the relevance.

Screenshots

Link to pull request in documentation repositories

User Documentation: home-assistant/companion.home-assistant#

Developer Documentation: home-assistant/developers.home-assistant#

Any other notes

[TimoPtr]

@brownard I want to thank you again for your project that speed up a lot this first implementation on the official companion app. If you want to take a look or even contribute you are more than welcome.

By being in the companion app, the authentication is already handle and we could also expose some commands to start/stop listening to save battery.

[brownard]

I'm glad my work has been useful. Ultimately the companion app is the best place for this so it's great that it's been picked up here.

[synesthesiam]

Looks good as a first pass! I would suggest at some point loading the settings for each wake word model from their respective config files, e.g. https://github.com/OHF-Voice/micro-wake-word/releases/download/okay_nabu_20241226.3/okay_nabu.json

You may also wake to include options for adjust the wake word sensitivity: https://github.com/esphome/home-assistant-voice-pe/blob/a379b8c5c1a35eeebc8f9925c19aab68743517a4/home-assistant-voice.yaml#L1775

[jpelgrom]

Nice to see how the local wake word work pays off, so much cleaner than when I worked on this back when Assist was new.

What's the idea behind the microfrontend name? 'Frontend' suggest the HA frontend user interface to me which is unrelated.

Should the service ignore input while AssistActivity is listening?

If you're going ahead with this setup, don't forget to update dev docs to indicate you need to install CMake for building (at least it didn't do so automatically for me on Windows - after that building worked fine!).

[TimoPtr]

What's the idea behind the microfrontend name? 'Frontend' suggest the HA frontend user interface to me which is unrelated.

It's the name of the TFlite lib we use https://github.com/tensorflow/tflite-micro/tree/main/tensorflow/lite/experimental/microfrontend.

Should the service ignore input while AssistActivity is listening?

From my initial test it seems that the OS is simply stop feeding our service when something on top is asking to the mic. I tried for instance with taking a video and it did not trigger the wake word detection.

[TimoPtr]

Merging this would have an impact on the size of the app without adding anything since the feature is only available through the DevPlayground, but the following PR will expose the feature properly from the settings.

[github-actions]

Test Results

  159 files    159 suites   10m 21s ⏱️
1 208 tests 1 208 ✅ 0 💤 0 ❌
1 251 runs  1 243 ✅ 8 💤 0 ❌

Results for commit 011ca85.

♻️ This comment has been updated with latest results.

[jpelgrom]

There are debug log statement for almost all override functions, we probably want to reduce that.

For C++ code with source in ESPHome/OHF-Voice: how are we going to keep those updated when changes are made?

[jpelgrom]

Do you want to merge with TODOs or create issues / keep this feature only for debug (if only for debug it needs a check here)?

[TimoPtr]

To make it easier this time I would like to go with merging those, especially because I've addressed most of them already in the follow up PR and removed it there.

[TimoPtr]

There are debug log statement for almost all override functions, we probably want to reduce that.

For C++ code with source in ESPHome/OHF-Voice: how are we going to keep those updated when changes are made?

The C++ code within the repo is a very thin layer on top of TFLite micro code that's why compare to AVA I've decided to not embedded the source but instead download it.

The part that we might need actually to keep in sync is the small Kotlin layer that calls TF and the microfrontend, that kinda act as an orchestrator. Unfortunately at the moment we need to do this manually. This is already the case also for other projects like the linux micro wake word. Ultimately this layer is quite small and should be easy to maintain. The most complicated part is the model itself where we do use TF.

I'll do some cleanup in the logs 👍🏻 it was very useful to debug.

[TimoPtr]

In the end after running the detection I think the logs are still relevant, especially in the early stage of this feature. I don't log anything in the hot-path so it's not spamming the logs IMO.

[jpelgrom]

Functionally, this mostly seems to work beautifully. Not 100% sure restarting in the background works as expected but I suppose we'll find out during more live testing.

Testing showed that while it is possible to manually start the service, this won't work when not set as the default assistant app in the background.

2026-02-04 21:58:50.205 22365-22437 AssistVoic...ionService io....stant.companion.android.debug  I  Wake word 'Hey Jarvis' detected, launching Assist
2026-02-04 21:58:50.218  1849-13320 ActivityTaskManager     system_server                        E  Background activity launch blocked! goo.gle/android-bal [callingPackage: io.homeassistant.companion.android.debug; callingPackageTargetSdk: 36; callingUid: 10439; callingPid: 22365; appSwitchState: 1; callingUidHasVisibleActivity: false; callingUidHasVisibleNotPinnedActivity: false; callingUidHasNonAppVisibleWindow: false; callingUidProcState: FOREGROUND_SERVICE; isCallingUidPersistentSystemProcess: false; allowBalExemptionForSystemProcess: false; intent: Intent { flg=0x10000000 xflg=0x4 cmp=io.homeassistant.companion.android.debug/io.homeassistant.companion.android.assist.AssistActivity }; callerApp: ProcessRecord{7b79a44 22365:io.homeassistant.companion.android.debug/u0a439}; inVisibleTask: false; balAllowedByPiCreator: BSP.ALLOW_BAL; resultIfPiCreatorAllowsBal: BAL_BLOCK; callerStartMode: MODE_BACKGROUND_ACTIVITY_START_SYSTEM_DEFINED; hasRealCaller: true; isCallForResult: false; isPendingIntent: false; autoOptInReason: notPendingIntent; realCallingPackage: io.homeassistant.companion.android.debug; realCallingPackageTargetSdk: 36; realCallingUid: 10439; realCallingPid: 22365; realCallingUidHasVisibleActivity: false; realCallingUidHasVisibleNotPinnedActivity: false; realCallingUidHasNonAppVisibleWindow: false; realCallingUidProcState: FOREGROUND_SERVICE; isRealCallingUidPersistentSystemProcess: false; originatingPendingIntent: null; realCallerApp: ProcessRecord{7b79a44 22365:io.homeassistant.companion.android.debug/u0a439}; realInVisibleTask: false; balAllowedByPiSender: BSP.ALLOW_BAL; resultIfPiSenderAllowsBal: BAL_BLOCK; realCallerStartMode: MODE_BACKGROUND_ACTIVITY_START_SYSTEM_DEFINED; balDontBringExistingBackgroundTaskStackToFg: true]

As previously mentioned I still think there are slightly too many debug statements in the log for production, but given that it isn't going anywhere users can use it yet it's good enough for now.

I cannot really validate the more complicated detection code/C++ code (due to my inexperience), but it seems to work and match the projects you've mentioned reasonably well.

[jpelgrom]

You aren't adding the service in the manifest for Automotive, which is what makes the service work consistently. Shouldn't this be in the app module specific dependencies instead?

[TimoPtr]

I think I want the service in automotive to test how it behaves no?

[TimoPtr]

Could you tell or create an issue about why we don't have assist offered as default in automotive?

[TimoPtr]

https://source.android.com/docs/automotive/voice/voice_interaction_guide/app_development It seems we might need another kind of service for automotive but it doesn't seems impossible.

[TimoPtr]

if I move this in app only then I need to move the code in a module that is not added to automotive. I would like to avoid this for this iteration.

[jpelgrom]

Could you tell or create an issue about why we don't have assist offered as default in automotive?

The Automotive build is distributed to a couple of car manufacturers, and their reviews started asking for specific UI customizations and behaviors we had to implement to pass their policy. At the same time Assist simply isn't optimized for hands-free usage (like a button to trigger answer/response, and always voice only responses), so with the limited resources we have we decided to remove it in #4965, discussion directly leading up to it.

I also think but am not 100% sure I remember correctly that assistant apps with a voice service on Automotive need a more general voice recognition service? Step 6 from your URL sounds like it.

(Personally I also don't see why you'd want HA as your only digital assistant app in a car.)

[TimoPtr]

Testing showed that while it is possible to manually start the service, this won't work when not set as the default assistant app in the background.

Indeed in the next PR I've disabled the toggle when not set as default in the settings.

[TimoPtr]

Nice to see how the local wake word work pays off, so much cleaner than when I worked on this back when Assist was new.

What's the idea behind the microfrontend name? 'Frontend' suggest the HA frontend user interface to me which is unrelated.

Should the service ignore input while AssistActivity is listening?

If you're going ahead with this setup, don't forget to update dev docs to indicate you need to install CMake for building (at least it didn't do so automatically for me on Windows - after that building worked fine!).

Doc updated home-assistant/developers.home-assistant#2946

[jpelgrom]

Good enough as a base so far, let's refine in follow up PRs before making it accessible for end users.