Accurate 3D Hand Pose Estimation for Whole-Body 3D Human Mesh Estimation (Pose2Pose codes)

High-resolution video link: here

Introduction

This repo is official PyTorch implementation of Accurate 3D Hand Pose Estimation for Whole-Body 3D Human Mesh Estimation (CVPRW 2022). This repo contains body-only, hand-only, and face-only codes of the Pose2Pose. The whole-body codes of the Hand4Whole are available at here.

Quick demo

• Slightly change torchgeometry kernel code following here.
• Download the pre-trained Pose2Pose from any of here (body), here (hand), and here (face).
• Prepare input.png and pre-trained snapshot at any of demo/body, demo/hand, and demo/face folders.
• Prepare human_model_files folder following below Directory part and place it at common/utils/human_model_files.
• Go to any of demo/body, demo/hand, and demo/face folders and edit bbox.
• Run python demo.py --gpu 0.
• If you run this code in ssh environment without display device, do follow:

1、Install oemesa follow https://pyrender.readthedocs.io/en/latest/install/
2、Reinstall the specific pyopengl fork: https://github.com/mmatl/pyopengl
3、Set opengl's backend to egl or osmesa via os.environ["PYOPENGL_PLATFORM"] = "egl"

Directory

Root

The ${ROOT} is described as below.

${ROOT}  
|-- data  
|-- demo
|-- main  
|-- output  
|-- common
|   |-- utils
|   |   |-- human_model_files
|   |   |   |-- smpl
|   |   |   |   |-- SMPL_NEUTRAL.pkl
|   |   |   |-- smplx
|   |   |   |   |-- MANO_SMPLX_vertex_ids.pkl
|   |   |   |   |-- SMPL-X__FLAME_vertex_ids.npy
|   |   |   |   |-- SMPLX_NEUTRAL.pkl
|   |   |   |   |-- SMPLX_to_J14.pkl
|   |   |   |-- mano
|   |   |   |   |-- MANO_LEFT.pkl
|   |   |   |   |-- MANO_RIGHT.pkl
|   |   |   |-- flame
|   |   |   |   |-- flame_dynamic_embedding.npy
|   |   |   |   |-- flame_static_embedding.pkl
|   |   |   |   |-- FLAME_NEUTRAL.pkl

• data contains data loading codes and soft links to images and annotations directories.
• demo contains demo codes.
• common contains kernel codes for Pose2Pose.
• main contains high-level codes for training or testing the network.
• output contains log, trained models, visualized outputs, and test result.
• human_model_files contains smpl, smplx, mano, and flame 3D model files. Download the files from [smpl] [smplx] [mano] [flame].

Data

You need to follow directory structure of the data as below.

${ROOT}  
|-- data  
|   |-- AGORA
|   |   |-- data
|   |   |   |-- AGORA_train.json
|   |   |   |-- AGORA_validation.json
|   |   |   |-- AGORA_test_bbox.json
|   |   |   |-- images_1280x720
|   |   |   |-- images_3840x2160
|   |   |   |-- smplx_params_cam
|   |   |   |-- cam_params
|   |-- FFHQ
|   |   |-- FFHQ_FLAME_NeuralAnnot.json
|   |   |-- FFHQ.json
|   |-- FreiHAND
|   |   |-- data
|   |   |   |-- training
|   |   |   |-- evaluation
|   |   |   |-- freihand_train_coco.json
|   |   |   |-- freihand_train_data.json
|   |   |   |-- freihand_eval_coco.json
|   |   |   |-- freihand_eval_data.json
|   |-- Human36M  
|   |   |-- images  
|   |   |-- annotations   
|   |-- InterHand26M
|   |   |-- images
|   |   |-- annotations
|   |-- MPII
|   |   |-- data
|   |   |   |-- images
|   |   |   |-- annotations
|   |-- MPI_INF_3DHP
|   |   |-- data
|   |   |   |-- images_1k
|   |   |   |-- MPI-INF-3DHP_1k.json
|   |   |   |-- MPI-INF-3DHP_camera_1k.json
|   |   |   |-- MPI-INF-3DHP_joint_3d.json
|   |   |   |-- MPI-INF-3DHP_SMPL_NeuralAnnot.json
|   |-- MSCOCO  
|   |   |-- images  
|   |   |   |-- train2017  
|   |   |   |-- val2017  
|   |   |-- annotations 
|   |-- PW3D
|   |   |-- data
|   |   |   |-- 3DPW_train.json
|   |   |   |-- 3DPW_validation.json
|   |   |   |-- 3DPW_test.json
|   |   |-- imageFiles

• 3D body datasets: AGORA, Human3.6M, MPII, MPI-INF-3DHP, MSCOCO, and 3DPW
• 3D hand datasets: AGORA, FreiHAND, and MSCOCO
• 3D face datasets: AGORA, FFHQ, and MSCOCO
• Refer to AGORA [parsing codes]
• Download FFHQ parsed data and FLAME parameters [data][FLAME parameters from NeuralAnnot]
• Download FreiHAND parsed data [data] [bbox]
• Download Human3.6M parsed data and SMPL parameters [data][SMPL parameters from NeuralAnnot]
• Download InterHand2.6M dataset from here.
• Download MPII parsed data and SMPL parameters [data][SMPL parameters from NeuralAnnot]
• Download MPI-INF-3DHP parsed data and SMPL parameters [data][SMPL parameters from NeuralAnnot]
• Download MSCOCO SMPL/MANO/FLAME parameters [SMPL parameters from NeuralAnnot] [MANO parameters from NeuralAnnot] [FLAME parameters from NeuralAnnot]
• Download 3DPW parsed data [data]
• All annotation files follow MSCOCO format. If you want to add your own dataset, you have to convert it to MSCOCO format.

Output

You need to follow the directory structure of the output folder as below.

${ROOT}  
|-- output  
|   |-- log  
|   |-- model_dump  
|   |-- result  
|   |-- vis  

• Creating output folder as soft link form is recommended instead of folder form because it would take large storage capacity.
• log folder contains training log file.
• model_dump folder contains saved checkpoints for each epoch.
• result folder contains final estimation files generated in the testing stage.
• vis folder contains visualized results.

Running Pose2Pose

• In the main/config.py, you can change datasets to use.

Train

In the main folder, run

python train.py --gpu 0-3 --parts body

to train body-only Pose2Pose on the GPU 0,1,2,3. --gpu 0,1,2,3 can be used instead of --gpu 0-3. You can chnage body to hand or face for the hand-only and face-only Pose2Pose, respectively. To train body-only Pose2Pose from the pre-trained 2D human pose estimation network, download this and place it at tool folder. Then run python convert_simple_to_pose2pose.py. Finally, move the generated snapshot_0.pth.tar to output/model_dump. Then, run

python train.py --gpu 0-3 --parts body --continue

Test

Place trained model at the output/model_dump/.

In the main folder, run

python test.py --gpu 0-3 --parts body --test_epoch 6

to test body-only Pose2Pose on the GPU 0,1,2,3 with60th epoch trained model. --gpu 0,1,2,3 can be used instead of --gpu 0-3. You can chnage body to hand or face for the hand-only and face-only Pose2Pose, respectively.

Models

• Download body-only Pose2Pose trained on H36M+MSCOCO+MPII from here.
• Download body-only Pose2Pose fine-tuned on AGORA from here.
• Download hand-only Pose2Pose trained on FreiHAND+InterHand2.6M+MSCOCO from here.
• Download face-only Pose2Pose trained on FFHQ+MSCOCO from here.

Results

3D body-only and hand-only results

Troubleshoots

• RuntimeError: Subtraction, the '-' operator, with a bool tensor is not supported. If you are trying to invert a mask, use the '~' or 'logical_not()' operator instead.: Go to here

Reference

@InProceedings{Moon_2022_CVPRW_Hand4Whole,  
author = {Moon, Gyeongsik and Choi, Hongsuk and Lee, Kyoung Mu},  
title = {Accurate 3D Hand Pose Estimation for Whole-Body 3D Human Mesh Estimation},  
booktitle = {Computer Vision and Pattern Recognition Workshop (CVPRW)},  
year = {2022}  
}