diffphys

This repo contains the differentiable physics simulation module in "PPR: Physically Plausible Reconstruction from Monocular Videos". It performs motion imitation given a target trajectory by optimizing control reference, PD gains, body mass, global se3, and initial velocity. For instructions on combining differentiable rendering with physics, please take a look at PPR.

[Project page]

Installation

Create a clean conda environment (skip if you have installed the lab4d environment)

mamba create -n ppr-diffphys python=3.9

Install pytorch. Replace mamba with conda if mamba is not installed

conda activate ppr-diffphys
mamba install pytorch==2.0.0 torchvision==0.15.0 torchaudio==2.0.0 pytorch-cuda=11.7 -c pytorch -c nvidia

Install cudatoolkit-dev with a version matching pytorch. Skip if it is previously installed.

mamba install -c conda-forge cudatoolkit-dev==11.7

Then install dependencies:

cd ppr-diffphys
CUDA_HOME=$CONDA_PREFIX pip install -r requirements.txt
pip install urdfpy==0.0.22 --no-deps

Replace with CUDA_HOME=/path-to-cuda-root/ if cuda is installed elsewhere, not inside the current conda env.

Motion Imitation on Mocap Data

To get results on Mocap data derived from motion_imitation, execute

bash run.sh

The results will be stored in the following directory: logdir/mi-xx-0/.

Visualization at 0 iteration (left to right: target, simulated, control reference)

all-00000.mp4

Visualization at 100 iteration (left to right: target, simulated, control reference)

all-00100.mp4

To generate additional visualizations over iterations, execute:

python render_intermediate.py --testdir logdir/mi-pace-0/ --data_class sim

sim.mp4

DiffRen+DiffSim

Implemented at lab4d@ppr. Please see PPR for instructions.

Citation

If you find this repository useful for your research, please cite the following work.

@inproceedings{yang2023ppr,
	title={Physically Plausible Reconstruction from Monocular Videos},
	author={Yang, Gengshan
	and Yang, Shuo
	and Zhang, John Z.
	and Manchester, Zachary
	and Ramanan, Deva},
	booktitle = {ICCV},
	year={2023},
}

Acknowledgement

• The differentiable physics simulation (e.g., kinematics, dynamics, contact) uses Warp internally.
• The laikago robot URDF file is taken from TDS, the quadruped URDF file is converted and modified from Mode-Adaptive Neural Networks for Quadruped Motion Control, and the human URDF file is converted from Mujoco.