[REQ] AI-dubbing

[MarcoRavich]

Hi there,
inspired by the AI Adventure Game Assistant project that I've just discovered, i think that it would be useful to combine some already available functions (demucs/whisper) with a TTS and a voice clone to regenerate a new voice track into another language.

As always I've already collected "some" - hopefully inspiring - open resources for the HyMPS project \ AI-based \ Voicing \ Dubbers.

Thanks in advance !

[RyanMetcalfeInt8]

Hi @MarcoRavich,

Thanks for the inspiration, as usual!

OpenVINO GenAI project actually just added some new preview features for text-to-speech via microsoft/speecht5_tts: https://github.com/openvinotoolkit/openvino.genai/tree/master/samples/python/speech_generation

And since we're already migrating our whisper features to use OpenVINO GenAI, it wouldn't be too much work for us to add support for this to Audacity.

I'm thinking that the feature could look something like this. A user would be able to generate speech from a text-box, or from a label track (i.e. output from whisper transcription). Something like that.

Edit: I want to also support the voice clone feature, by allowing user to select some small segment of speech. Looks like this one will require some python -> C++ ports from the speechbrain logic... but hopefully not a big deal.

Any ideas around improved usability, look & feel are welcome :)

Thanks,
Ryan

[MarcoRavich]

Hi @MarcoRavich,

Thanks for the inspiration, as usual!

Hi Ryan, sorry for the delay in replying, but I was busy for a (far) music festival audio/video recordings this weekend.

OpenVINO GenAI project actually just added some new preview features for text-to-speech via microsoft/speecht5_tts: https://github.com/openvinotoolkit/openvino.genai/tree/master/samples/python/speech_generation

And since we're already migrating our whisper features to use OpenVINO GenAI, it wouldn't be too much work for us to add support for this to Audacity.

I didn't know about @intel's GenAI project, sounds interesting!
I have to open a "request issue" to ask @RunDevelopment / @joeyballentine to support it in chaiNNer...

I'm thinking that the feature could look something like this. A user would be able to generate speech from a text-box, or from a label track (i.e. output from whisper transcription). Something like that.

Well, I'm not an "automatic dubbing" technicalities expert honestly, but to me the @kevinrss01's Dubbing Engine scheme summarize all the needed steps (in your case refer only to the audio part, of course) well:

An alternative (but less complete, to me) can be the @HallowSiddharth's Voice Craft AI one:

Last but not least, I've also realized that @iGerman00's Pollyduble is a bit different from the rest 'cause it exploits @haoheliu's Voicefixer restoration too...

Edit: I want to also support the voice clone feature, by allowing user to select some small segment of speech. Looks like this one will require some python -> C++ ports from the speechbrain logic... but hopefully not a big deal.

Well, as you know there are specialized cloners but I suggest to check the interesting AI-Powered Speech Synthesis features desctiprion in the Linly-Dubbing repo...

Any ideas around improved usability, look & feel are welcome :)

I'm still not entirely convinced myself that it's really feasible to implement all these features into single Audacity plugin (it's probably better to have a dedicated standalone GUI), but here are other's open source dubbers (available screenshots) GUIs:

• @DigitalEpidemic's AI Voice Translator:
• @jianchang512's pyvideotrans:
• @abus-aikorea's Kara-Audio:
• @Kedreamix's Linly-Dubbing:

As you can see, almost all have multiple tabs to manage the various functions, but perhaps it is also possible to do it with a sequential wizard.

Thanks, Ryan

Thank you for your friendliness for the open source community!