Course group project for Digital Humans, 263-5806-00L, ETH Zürich, FS2025.
This project aims to integrate VolumetricSMPL into DartControl. The human model achieves a volumetric representation and enables 3D scenes and access to collision loss terms.
Text-conditioned human motion generation allows for intuitive control over digital humans via natural language. The recent framework DART enables real-time motion generation through latent diffusion and autoregressive control, also allowing goal-reaching motion. However, this model sometimes results in physically implausible results and collisions with the surrounding scene. To address this limitation, we present an extension of DART, incorporating the volumetric body model VolumetricSMPL. We leverage a signed distance field representation of the body model and incorporate it into the given DART framework. We show that our extension reduces collisions by more than 50% on average while maintaining adherence to high-level textual goals. Ablation studies further analyze the trade-offs introduced by volumetric constraints. Our results demonstrate that incorporating volumetric body geometry significantly improves scene-aware motion generation in complex 3D environments.
This section describes the setup and installation for the code of the group project. See the description of DART (Getting Started) for the complete setup. This section is structured into three subsections: Requirements and Environment, Download Project Repository, and Download Data and Model Checkpoints.
The experimental setup operated on a conda (v.25.1.1) environment with Python (v.3.12.9) on Ubuntu (v.22.04.5). We recommend to use miniconda (Miniconda - ANACONDA). The local device used 2 Intel Xeon CPUs and 1 NVIDIA GTX 1080 Ti from the student cluster.
conda env create -f environment.yml
conda activate iDART
Run the following commands in your console. The command will download the project code and activate the external dependencies.
git clone git@github.com:seba-heck/iDART.git
cd iDART
The user need to connect with GitHub SSH (Secure Shell Protocol). For the guideline, see Generating a new SSH key and adding it to the ssh-agent.
The project depends on model checkpoints and data sets from DART and data for the body models. Please follow the links, download the material, unpack and merge it with this repository.
- DART data - Google Drive: folders can be copied into the root directory.
- SMPL-X body model: insert into data folder (exact structure below)
- SMPL-H body model: insert into smplx folder (exact structure below)
- EgoBody Dataset: download scene dataset here (not necessary, point clouds available in
data/scene/)
Root folder structure
.
├── config_files
├── control
├── data *new
├── data_loaders
├── data_scripts
├── demos
├── diffusion
├── environment.yml
├── evaluation
├── misc
├── mld
├── mld_denoiser *new
├── model
├── mvae *new
├── policy_train *new
├── README.md
├── scenes
├── utils
└── visualize
...
Data folder structure
data
├── action_statistics.json
├── fps_dict_all.json
├── fps_dict.json
├── hml3d_smplh
│ └── seq_data_zero_male
├── inbetween
│ └── pace_in_circles
├── joint_skin_dist.json
├── optim_interaction
│ ├── climb_down.json
│ └── sit.json
├── scenes
│ └── demo
├── seq_data_zero_male
│ ├── mean_std_h1_f1.pkl
│ ├── mean_std_h2_f16.pkl
│ ├── mean_std_h2_f8.pkl
│ ├── train_text_embedding_dict.pkl
│ └── val_text_embedding_dict.pkl
├── smplx_lockedhead_20230207 *from other source
│ └── models_lockedhead *unpack and move models here
├── stand_20fps.pkl
├── stand.pkl
├── test_locomotion
│ ├── demo_walk.json
│ ├── random.json
│ ├── test_hop_long.json
│ ├── test_run_long.json
│ └── test_walk_long.json
└── traj_test
├── dense_frame180_walk_circle
├── dense_frame180_wave_right_hand_circle
├── sparse_frame180_walk_square
└── sparse_punch
SMPL-X folder structure
data/smplx_lockedhead_20230207/
└── models_lockedhead
├── smplh *from MANO/SMPL-H
│ ├── female
│ ├── info.txt
│ ├── LICENSE.txt
│ ├── male
│ └── neutral
└── smplx *from SMPL-X
├── md5sums.txt
├── SMPLX_FEMALE.npz
├── SMPLX_MALE.npz
└── SMPLX_NEUTRAL.npz
Use the correct names for the folders, especially for the SMPl-X folder, and be careful with data folder, it already contains some necessary files.
The folder demos/ contains several scripts which run some examples and experiments. The output motions are saved in mld_denoiser/mld_fps_clip_repeat_euler/checkpoint_300000/optim/. These can be visualised with Blender, see Visualisation - DART. Some examples create a video of the SDF in bin/results/. Following are some instructions for the demos.
Examples of different motions and scenes.
source demos/examples.sh
Experiments with Mesh2SDF and VolSMPL.
source demos/mesh2sdf.sh
- Liza Polupanova, D-MATH, ipolupanova@student.ethz.ch
- David Blickenstorfer, D-MATH, davidbl@student.ethz.ch
- Sebastian Heckers, D-MATH, sebastian.heckers@inf.ethz.ch
The authors thanks Kaifeng Zhao for his supervising during the project.
The code was developed at ETH Zurich and is part of the course: Digital Humans, 263-5806-00L.