Speech recognition is English-only while using multilingual backend/s

[mvidelatraduc]

Both remote (Wisprflow) and local (Whisper-XXX) speech recognition assume English language speech as input. I'm fairly certain that the models are their multilingual variants and not the ".en" ones: the debug mode shows them as "whisper-XXX: multi". From what I know about Whisper, it'd be a matter of passing a flag (ideally, by selecting a language in the app).

I'm unaware of how this would work for Wisprflow, but they advertise "over 100" languages, so presumably the process would be similar.

PS It'd be good to add to the documentation whether Wisprflow is configured with the private/no data retention mode activated, and whether users will be able to use their own API keys if need be in the future.

[idavydov]

To add: being able to e.g., respond to messages in more than a single language would be amazing.

I think this was almost working in the past (though, output parsing broke down on some characters). I would love to see some kind of multi-language support in the future. For now it just doesn't detect any speech

[mvidelatraduc]

Maybe some sort of extra action in the notification like "Answer in Lang. 1"/"Answer in Lang. 2" for lack of good detection? (Not sure how that would be implemented in the app, or in the watches both old and new for that matter.)

In any case, as you've said, the minimum for now would be selecting languages other than English as a global setting to achieve parity with the old app.

[idavydov]

I briefly looked at the whisper (python) API. In principle there are two relevant calls.

1. There's model.detect_language(mel) which gives you probabilities.
2. There's whisper.DecodingOptions(language="en"), where you can set the language.

So one approach would be to set allowed languages in settings. And then select language with the maximum probability (among the ones enabled in the app) for decoding.

[annando]

For me (App Version 1.0.6.9 on Android), the "on device" recognition only seems to recognize English (when I speak something in German, it is detected as something similar sounding in English. The remote detection seems to work fine for German.

[mvidelatraduc]

After some additional testing, longer sentences get detected as Spanish, but short to medium ones don't (one even got detected as what I assume is Finnish).