more_hubert_cases_audio_only.md

More hubert cases (Audio-only Model)

Features of this model include:

• The driving signals require only one image plus an audio segment.
• It offers good visual stability.
• It forces the face to orient forward.
• The expressiveness is moderate.

Einstein

python ./code/demo.py \
    --infer_type 'hubert_audio_only' \
    --stage1_checkpoint_path 'ckpts/stage1.ckpt' \
    --stage2_checkpoint_path 'ckpts/stage2_audio_only_hubert.ckpt' \
    --test_image_path 'test_demos/portraits/Einstein.png' \
    --test_audio_path 'test_demos/audios/english_male.mp3' \
    --test_hubert_path 'test_demos/audios_hubert/english_male.npy' \
    --result_path 'outputs/Einstein_hubert/' 

• The generated video of this sample will be saved to outputs/Einstein_hubert/Einstein-english_male.mp4.

Storytelling (Chinese)

python ./code/demo.py \
    --infer_type 'hubert_audio_only' \
    --stage1_checkpoint_path 'ckpts/stage1.ckpt' \
    --stage2_checkpoint_path 'ckpts/stage2_audio_only_hubert.ckpt' \
    --test_image_path 'test_demos/portraits/girl.png' \
    --test_audio_path 'test_demos/audios/lianliru.wav' \
    --test_hubert_path 'test_demos/audios_hubert/lianliru.npy' \
    --result_path 'outputs/lianliru_hubert/' 

• The generated video of this sample will be saved to outputs/lianliru_hubert/girl-lianliru.mp4.

Long Story Generation

python ./code/demo.py \
    --infer_type 'hubert_audio_only' \
    --stage1_checkpoint_path 'ckpts/stage1.ckpt' \
    --stage2_checkpoint_path 'ckpts/stage2_audio_only_hubert.ckpt' \
    --test_image_path 'test_demos/portraits/cartoon_girl.png' \
    --test_audio_path 'test_demos/audios/mars.wav' \
    --test_hubert_path 'test_demos/audios_hubert/mars.npy' \
    --result_path 'outputs/cartoon_girl_mars_story_hubert/' 

• The generated video of this sample will be saved to outputs/cartoon_girl_mars_story_hubert/cartoon_girl-mars.mp4.

Statue

python ./code/demo.py \
    --infer_type 'hubert_audio_only' \
    --stage1_checkpoint_path 'ckpts/stage1.ckpt' \
    --stage2_checkpoint_path 'ckpts/stage2_audio_only_hubert.ckpt' \
    --test_image_path 'test_demos/portraits/statue.jpg' \
    --test_audio_path 'test_demos/audios/statue.wav' \
    --test_hubert_path 'test_demos/audios_hubert/statue.npy' \
    --result_path 'outputs/statue_hubert/' 

• The generated video of this sample will be saved to outputs/statue_hubert/statue-statue.mp4.

Your own case

python ./code/demo.py \
    --infer_type 'hubert_audio_only' \
    --stage1_checkpoint_path 'ckpts/stage1.ckpt' \
    --stage2_checkpoint_path 'ckpts/stage2_audio_only_hubert.ckpt' \
    --test_image_path '/path/to/image_path(png or jpg)' \
    --test_audio_path '/path/to/audio_path(wav or mp3)' \
    --test_hubert_path '/path/to/hubert_feature_path(npy)' \
    --result_path '/path/to/saved_folder/' 

Change the above /path/to/xxx to your own path or folder.

Reference

• Image of Einstein.png is from GAIA
• 'lianliru.wav' is from StoryTTS dataset.
• Image of girl.png and cartoon_girl.png were generated by yiyan.