Detect multiple languages in the same audio

[matsumurae]

I open this as an idea but I was curious to know if this would be possible. It's not a common issue (I believe) but I'm trilingual so I tend to use multiple languages on the same sentence. I was just thinking about "would be this possible?".

We can choose the language or do auto-detect so, would auto-detect possible to run into this?

[chrwiencke]

It is a good idea and something I would like to see as well. You could run a language check every time it is uncertain about the words, maybe.

[bvpotapenko]

Would be glad to see it working, too!
I work on a voice assistant, and one of typical user cases is when a user asks to translate from English into their first language, so the utterance consists of words in both languages.

And multilingual model fails, it tries to interpret sounds of the first language as English which ends up as gibberish.

For example:
User says: "用汉语怎么说 a big black cat on a chair"
Whisper.cpp transcribes: "It is your unit, the martial art, a big black cat on a chair."

If there is a known solution for this problem, I'd appreciate some advice! Thank you!

[bvpotapenko]

Somehow solved it by:

1. switching from a "base" model to a "medium-q5";
2. using a prompt: "Hello 你好";
3. setting language to: "Auto".