This is the code for the paper
Anthony Bourached, Ryan-Rhys Griffiths, Robert Gray, Ashwani Jha, Parashkev Nachev. Generative Model-Enhanced Human Motion Prediction. Under review at ICLR 2021. Accepted at NeurIPS workshop on Interpretable Inductive Biases and Physically Structured Learning.
Some older versions may work. But we used the following:
- cuda 10.1
- Python 3.6.9
- Pytorch 1.6.0
- progress 1.5
Human3.6m in exponential map can be downloaded from here.
CMU mocap was obtained from the repo of ConvSeq2Seq paper.
All the running args are defined in opt.py. We use following commands to train on different datasets and representations. To train on angle space, in-distribution, H3.6M:
python3 main.py --data_dir "[Path To Your H36M data]/h3.6m/dataset/" --variational --lambda 0.003 --n_z 8 --dropout 0.3 --lr_gamma 1.0 --input_n 10 --output_n 10 --dct_n 20in-distribution (CMU):
python3 main.py --dataset 'cmu_mocap' --data_dir "[Path To Your CMU data]/cmu_mocap/" --variational --lambda 0.003 --n_z 8 --dropout 0.3 --lr_gamma 1.0 --input_n 10 --output_n 25 --dct_n 35to train on 3D space for CMU, simply change the --dataset 'cmu_mocap' to --dataset 'cmu_mocap_3d. This flag is 'h3.6m' by default.
To train on 'walking' and test out-of-distribution (for h3.6M), include the extra flag:
--out_of_distribution 'walking' identically to train on 'basketball' and test out-of-distribution (for CMU), include the extra flag:
--out_of_distribution 'basketball' The same models may be trained (or used for inference independent of how they were trained) without the VGAE branch by removing the
--variational
flag.
python3 hyperparameter_search.py --num_trials 10 --epoch 100 --variational
Inference on latent spaces for trained model, saves to latents.csv (also save DCT inputs, to inputs.csv)
python3 interpretability.py --dataset 'cmu_mocap' --model_path "[Path To Your Trained Model].pth.tar"If you use our code, and/or build on our work, please cite our paper:
@article{https://doi.org/10.1002/ail2.63,
author = {Bourached, Anthony and Griffiths, Ryan-Rhys and Gray, Robert and Jha, Ashwani and Nachev, Parashkev},
title = {Generative Model-Enhanced Human Motion Prediction},
journal = {Applied AI Letters},
volume = {n/a},
number = {n/a},
pages = {},
doi = {https://doi.org/10.1002/ail2.63},
url = {https://onlinelibrary.wiley.com/doi/abs/10.1002/ail2.63},
eprint = {https://onlinelibrary.wiley.com/doi/pdf/10.1002/ail2.63},
abstract = {Abstract The task of predicting human motion is complicated by the natural heterogeneity and compositionality of actions, necessitating robustness to distributional shifts as far as out-of-distribution (OoD). Here we formulate a new OoD benchmark based on the Human3.6M and CMU motion capture datasets, and introduce a hybrid framework for hardening discriminative architectures to OoD failure by augmenting them with a generative model. When applied to current state-of-theart discriminative models, we show that the proposed approach improves OoD robustness without sacrificing in-distribution performance. We suggest human motion predictors ought to be constructed with OoD challenges in mind, and provide an extensible general framework for hardening diverse discriminative architectures to extreme distributional shift. The code is available at https: //github.com/bouracha/OoDMotion.}
}
The codebase is built on that of https://github.com/wei-mao-2019/LearnTrajDep and depends heavily on their work in Learning Trajectory Dependencies for Human Motion Prediction (ICCV 2019), and History Repeats Itself: Human Motion Prediction via Motion Attention (ECCV 2020). Thus please also cite:
@inproceedings{wei2019motion,
title={Learning Trajectory Dependencies for Human Motion Prediction},
author={Wei, Mao and Miaomiao, Liu and Mathieu, Salzemann and Hongdong, Li},
booktitle={ICCV},
year={2019}
}
and
@article{mao2020history,
title={History Repeats Itself: Human Motion Prediction via Motion Attention},
author={Mao, Wei and Liu, Miaomiao and Salzmann, Mathieu},
journal={arXiv preprint arXiv:2007.11755},
year={2020}
}
MIT