Speech To Facial Animation Using GANs

This repo contains the pytorch implementation of achieving facial animation from given face image and speech input using Generative Adversarial Nets (See References).

Results

Some of the generated videos are found here.

Implementation

GAN setup

References

[1] Generative Adversarial Nets

@article{goodfellow2014generative,
  title={Generative adversarial networks},
  author={Goodfellow, Ian J and Pouget-Abadie, Jean and Mirza, Mehdi and Xu, Bing and Warde-Farley, David and Ozair, Sherjil and Courville, Aaron and Bengio, Yoshua},
  journal={arXiv preprint arXiv:1406.2661},
  year={2014}
}

[2] The Audio-Visual Lombard Grid Speech Corpus

@article{Alghamdi_2018,
	doi = {10.1121/1.5042758},
	url = {https://doi.org/10.1121%2F1.5042758},
	year = 2018,
	month = {jun},
	publisher = {Acoustical Society of America ({ASA})},
	volume = {143},
	number = {6},
	pages = {EL523--EL529},
	author = {Najwa Alghamdi and Steve Maddock and Ricard Marxer and Jon Barker and Guy J. Brown},
	title = {A corpus of audio-visual Lombard speech with frontal and profile views},
	journal = {The Journal of the Acoustical Society of America}
}

[3] Realistic Facial Animation using GANs

@article{Vougioukas_2019,
	doi = {10.1007/s11263-019-01251-8},
	url = {https://doi.org/10.1007%2Fs11263-019-01251-8},
	year = 2019,
	month = {oct},
	publisher = {Springer Science and Business Media {LLC}},
	volume = {128},
	number = {5},
	pages = {1398--1413},
	author = {Konstantinos Vougioukas and Stavros Petridis and Maja Pantic},
	title = {Realistic Speech-Driven Facial Animation with {GANs}},
	journal = {International Journal of Computer Vision}
}

[4] End to End Facial Animation using Temporal GANs

@article{vougioukas2018end,
  title={End-to-end speech-driven facial animation with temporal gans},
  author={Vougioukas, Konstantinos and Petridis, Stavros and Pantic, Maja},
  journal={arXiv preprint arXiv:1805.09313},
  year={2018}
}

[5] A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild

@inproceedings{10.1145/3394171.3413532,
author = {Prajwal, K R and Mukhopadhyay, Rudrabha and Namboodiri, Vinay P. and Jawahar, C.V.},
title = {A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild},
year = {2020},
isbn = {9781450379885},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3394171.3413532},
doi = {10.1145/3394171.3413532},
booktitle = {Proceedings of the 28th ACM International Conference on Multimedia},
pages = {484–492},
numpages = {9},
keywords = {lip sync, talking face generation, video generation},
location = {Seattle, WA, USA},
series = {MM '20}
}