PhyRecon: Physically Plausible Neural Scene Reconstruction

NeurIPS 2024

Junfeng Ni*, Yixin Chen*, Bohan Jing, Nan Jiang, Bin Wang,
Bo Dai, Puhao Li, Yixin Zhu, Song-Chun Zhu, Siyuan Huang

PhyRecon harnesses both differentiable rendering and differentiable physics simulation to achieve physically plausible scene reconstruction from multi-view images.

News

• [2025-3-20] Our follow-up work, DP-Recon, which integrates generative priors into decompositional neural scene reconstruction, is now out! Check it out here!
• [2025/1/14] Code is released. For more information, please visit our project page!
• [2025/1/14] Our Differentiable-Rigid-Body-Simulator is open sourced!

Installation

conda create -n phyrecon python=3.9
conda activate phyrecon
pip install torch==2.0.1 torchvision==0.15.2 --index-url https://download.pytorch.org/whl/cu118
pip install -r requirements.txt

Data

Please download the preprocessed data and unzip in the data folder. The resulting folder structure should be:

└── PhyRecon
  └── data
    ├── replica
        ├── scan ...
    ├── scannet
        ├── scan ...
    ├── scannetpp
        ├── scan ...

Since the full dataset is quite large, we use the scannetpp/scan1 as an example scene. Please download this scene here and ensure it is placed under data/scannetpp following the folder structure.

Training

cd code
torchrun training/exp_runner.py --conf CONFIG  --scan_id SCAN_ID
# CONFIG is the config file in code/confs, and SCAN_ID is the id of the scene to reconstruct.
### for the example scene from scannetpp
# torchrun training/exp_runner.py --conf confs/scannetpp_phyrecon_grid.conf --scan_id 1
### By default, training logs are saved using wandb. If you prefer to use TensorBoard, you can set the '--none_wandb'. 

Evaluation

1. Reconstruction

# evaluate total scene
python eval/eval_scene_recon.py --dataset_type DATASET     # DATASET could be 'scannetpp', 'scannet' and 'replica'
# evaluate each object
python eval/eval_object_recon.py --dataset_type DATASET

2. Physical Stability

First, install Issac Gym by following the installation tutorial. Once installed, you can evaluate object stability using the following command:

# evaluate stability
python eval/eval_stable_ratio.py --dataset_type DATASET
### If you want to enable visualization and interact with the window, please set `--vis` flag.
### Issac Gym will load the VHACD cache for the same object, 
### If you need to evaluate the stable ratio for the same dataset and scene multiple times, please set the `--test_idx` according to the number of tests you are running.

3. Performance

The performance of PhyRecon suffers from some instability and randomness due to the following reasons:

• The gap between the particle-based simulator used in our method and the mesh-based simulator (Issac Gym) used for stability evaluation.
• The small floaters of the reconstructed mesh may greatly impact the results.
• Some objects lack suitable collision points due to insufficient observations.

To mitigate the impact of instability, we rerun our method three times and report the mean and standard deviation, as shown in the table below. The differences between these results and those reported in the paper stem from the following:

• The baseline methods, ObjectSDF++ and RICO, fail to reconstruct certain objects during training, which causes some objects to be missing. In this paper, we restricted our evaluation to objects that appear across all methods. Our reproduced results included all objects reconstructed using our method.
• For the stability evaluation in the paper, we used the reconstructed backgrounds, which are more suitable for reconstructed objects but require significant manual effort. Instead, we provided GT backgrounds for all the scenes in the dataset for more convenient evaluation in the future.

Our reconstructed results are available here.

Dataset		Scene Metrics			Object Metrics			Stable Metric
		CD(↓)	F-Score(↑)	NC(↑)	CD(↓)	F-Score(↑)	NC(↑)	SR(↑)
		ScanNet++	In Paper	3.34	81.53	90.10	3.28	87.21	86.16	78.16
Reproduced	3.66±0.06		80.87±0.92	88.24±0.45	2.92±0.11	88.07±0.34	82.98±0.58	71.84±1.46
ScanNet	In Paper	8.34	63.01	86.57	7.92	75.54	82.54	70.31
	Reproduced	8.56±0.25	58.22±3.89	83.29±1.02	5.55±0.34	71.43±2.18	80.28±1.18	57.79±3.99
Replica	In Paper	3.68	85.61	89.45	3.86	81.30	84.91	77.63
	Reproduced	3.66±0.06	83.13±1.71	92.06±1.57	3.45±0.19	85.91±1.47	87.29±1.03	66.69±3.20

Acknowledgements

Some codes are borrowed from MonoSDF, RICO and ObjectSDF++. We thank all the authors for their great work.

Citation

@inproceedings{ni2024phyrecon,
  title={PhyRecon: Physically Plausible Neural Scene Reconstruction}, 
  author={Ni, Junfeng and Chen, Yixin and Jing, Bohan and Jiang, Nan and Wang, Bin and Dai, Bo and Li, Puhao and Zhu, Yixin and Zhu, Song-Chun and Huang, Siyuan},
  journal={Advances in Neural Information Processing Systems},
  year={2024}
}