A VIDEO📽 TO VIDEO🎞 SPEECH TRANSLATOR WITH RETAINING SPEAKER VOICE🗣 AND LIPSYNC👄.
- The colab notebook project v1.2 has the full code. it's quite resource(memory and gpu) uitilizing.
- Projectv2-api uses replicate apis to reduce resource utilization.
We tried to make translator where you can UPLOAD A VIDEO HAVING ANY LANGUAGE TO PRODUCE SAME VIDEO WITH ENGLISH SPEAKING SPEAKER HAVING HIS OWN VOICE WITH LIPSYNC To make this toolkit.
Or add any others voice👀.
We used Open AI's Whisper AI
Tortoise-tts
Wav2lip
We don't own any of this above programs.
We seriallized these to produce resultant easier😇.
0. RUNTIME SHOULD BE ON GPU.
1. First audio file should be ~15 seconds long in english language.(in the voice to be cloned).
2. Upload non-english video with only single speaker.
3. Read the instructions and run each cell one by one.
In this version, You have upload the you voice to want to clone(speaker's english voice) to a website
we recommened huggingface.