Prosody-Enhanced Acoustic Pre-training and Acoustic-Disentangled Prosody Adapting for Movie Dubbing

[CVPR2025] Official implementation of paper "Prosody-Enhanced Acoustic Pre-training and Acoustic-Disentangled Prosody Adapting for Movie Dubbing"

🌼 Environment

Our python version is 3.8.18 and cuda version 11.8. It's possible to have another compatible version. Both training and inference are implemented with PyTorch on a GeForce RTX 4090 GPU.

conda create -n dubbing python=3.8.18
conda activate dubbing
pip install -r requirements.txt

🔧 Training

For First Stage (Acoustic Pre-training)

python train_first.py -p Configs/config_v2c_stage1.yml  # V2C-Animation benchmark
python train_first.py -p Configs/config_grid_stage1.yml  # GRID benchmark

For Second Stage (Prosody Adapting)

python train_second.py -p Configs/config_v2c.yml  # V2C-Animation benchmark
python train_second_grid.py -p Configs/config_grid.yml  # GRID benchmark

💡 Checkpoints

We provide the first stage and second stage pre-trained checkpoints on V2C-Animation and GRID benchmarks as follows, respectively:

First stage (For secnod stage only, can not directly generate wavform)

• V2C-Animation benchmark: Baidu Drive (b5wy), Google Drive.

• GRID benchmark: Baidu Drive (wj25), Google Drive

Second stage (Can used to directly generate waveform)

• V2C-Animation benchmark: Baidu Drive (3k4h), Google Drive.

• GRID benchmark: Baidu Drive (23vd), Google Drive

✍ Inference

For V2C-Animation Benchmark

There is three generation settings in V2C-Animation benchmark:

python inference_v2c.py -n 'YOUR_EXP_NAME' --epoch 'YOUR_EPOCH' --setting 1

python inference_v2c.py -n 'YOUR_EXP_NAME' --epoch 'YOUR_EPOCH' --setting 2

python inference_v2c.py -n 'YOUR_EXP_NAME' --epoch 'YOUR_EPOCH' --setting 3

For GRID benchmark

There is two generation settings in GRID benchmark:

python inference_grid.py -n 'YOUR_EXP_NAME' --epoch 'YOUR_EPOCH' --setting 1

python inference_grid.py -n 'YOUR_EXP_NAME' --epoch 'YOUR_EPOCH' --setting 2

📊 Dataset

• GRID (BaiduDrive (code: GRID) / GoogleDrive)
• V2C-Animation dataset (chenqi-Denoise2) (BaiduDrive (code: k9mb) / GoogleDrive)

🙏 Acknowledgments

We would like to thank the authors of previous related projects for generously sharing their code and insights: StyleTTS, StyleTTS2, StyleDubber, PL-BERT, and HiFi-GAN.

🤝 Citation

If you find our work useful, please consider citing:

@InProceedings{zhang_2025_produbber,
    author    = {Zhang, Zhedong and Li, Liang and Yan, Chenggang and Liu, Chunshan and van den Hengel, Anton and Qi, Yuankai},
    title     = {Prosody-Enhanced Acoustic Pre-training and Acoustic-Disentangled Prosody Adapting for Movie Dubbing},
    booktitle = {Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR)},
    month     = {June},
    year      = {2025},
    pages     = {172-182}
}