The Results about Audio to Video Generation Using AudioSet-Drum.

[Minglu58]

Hi,

Congratulations for your excellent work!

I am interested in using audio to generate video. I found that the scripts you provided for audio conditioning generation is just using audio only as the condition, not both audio and 15 frames as you say in paper. I tried to modify the scripts, like adding the audio latent indices and 15 latent frame indices, to generate the 16th frame and evaluate the 16th frame, but the SSIM and PSNR scores are not ideal actually. I was wondering whether my idea is reasonable and accurate, and whether you can kindly provide the scripts. I am also struggling to generate 45th frame using audio condition. Can it be gotten from sample_long_video.py scripts?

By the way, the way to calculating the SSIM and PSNR is suggested by the scripts from CCVS model. Could you please give me an idea on getting the final results close to the ones in paper?

Thank you for your help!

Minglu