This package contains code for the Deep Network for the Integrated 3D Sensing of Multiple People in Natural Images (MubyNet) paper published at the NeurIPS 2018 from Montreal, Canada.
By using the software, you are agreeing to the terms of the license agreement.
Our software is built on top of the Caffe deep learning library. The current version was developed by:
Andrei Zanfir , Elisabeta Marinoiu , Mihai Zanfir , Alin-Ionut Popa and Cristian Sminchisescu
Our model identifies human body structures (joints and limbs) in images, groups them based on 2d and 3d information fused using learned scoring functions, and optimally aggregates such responses into partial or complete 3d human skeleton hypotheses under kinematic tree constraints, but without knowing in advance the number of people in the scene and their visibility relations. We design a multi-task deep neural network with differentiable stages where the person grouping problem is formulated as an integer program based on learned body part scores parameterized by both 2d and 3d information. This avoids suboptimality resulting from separate 2d and 3d reasoning, with grouping performed based on the combined representation.
If you use this code/model for your research, please cite the following paper:
@inproceedings{mubynet_2018,
author = {Zanfir, Andrei and Marinoiu, Elisabeta and Zanfir, Mihai and Popa, Alin-Ionut and Sminchisescu, Cristian},
title = {Deep Network for the Integrated 3D Sensing of Multiple People in Natural Images},
booktitle = {Advances in Neural Information Processing Systems 31},
year = {2018}
}
First, clone the project by running:
git clone --recursive https://github.com/alinionutpopa/MubyNet.git
You need to compile the Caffe library in this repository. See http://caffe.berkeleyvision.org/installation.html for the latest installation instructions.
Check the users group in case you need help: https://groups.google.com/forum/#!forum/caffe-users
All done! Try our method!
First download the model that includes the trained weights from this link into the data/model folder. Also, change the caffepath variable from code/config_release.m file accordingly.
The MATLAB script for running the demo is demoMubyNet.m. Please note that the shape estimation module will be made available at a later time.
Contact: andrei.zanfir@imar.ro, elisabeta.marinoiu@imar.ro, mihai.zanfir@imar.ro, alin.popa@imar.ro, cristian.sminchisescu@math.lth.se
This work was supported in part by the European Research Council Consolidator grant SEED, CNCS-UEFISCDI (PN-III-P4-ID-PCE-2016-0535, PN-III-P4-ID-PCCF-2016-0180), the EU Horizon 2020 grant DE-ENIGMA (688835), and SSF.