PyTorch implementation for paper ShaDDR: Interactive Example-Based Geometry and Texture Generation via 3D Shape Detailization and Differentiable Rendering, Qimin Chen, Zhiqin Chen, Hang Zhou, Hao Zhang.
If you find our work useful in your research, please consider citing (to be updated):
@misc{chen2023shaddr,
title={ShaDDR: Real-Time Example-Based Geometry and Texture Generation via 3D Shape Detailization and Differentiable Rendering},
author={Qimin Chen and Zhiqin Chen and Hang Zhou and Hao Zhang},
year={2023},
eprint={2306.04889},
archivePrefix={arXiv}
}
Requirements:
- Python 3.7 with numpy, pillow, h5py, scipy, sklearn and Cython
- PyTorch 1.9 (other versions may also work)
- PyMCubes (for marching cubes)
- OpenCV-Python (for reading and writing images)
Build Cython module:
python setup.py build_ext --inplace
For data preparation, please see data_preparation.
We provide the ready-to-use datasets here.
We also provide the pre-trained network weights.
Make sure to train the geometry detailization first, then train texture generation.
python main.py --data_style style_color_car_16 --data_content content_car_train --data_dir ./data/02958343/ --alpha 0.2 --beta 10.0 --input_size 64 --output_size 512 --train --train_geo --gpu 0 --epoch 20
python main.py --data_style style_color_car_16 --data_content content_car_train --data_dir ./data/02958343/ --alpha 0.2 --beta 10.0 --input_size 64 --output_size 512 --train --train_tex --gpu 0 --epoch 20
python main.py --data_style style_color_plane_16 --data_content content_plane_train --data_dir ./data/02691156/ --alpha 0.1 --beta 10.0 --input_size 64 --output_size 512 --train --train_geo --gpu 0 --epoch 20
python main.py --data_style style_color_plane_16 --data_content content_plane_train --data_dir ./data/02691156/ --alpha 0.2 --beta 10.0 --input_size 64 --output_size 512 --train --train_tex --gpu 0 --epoch 20
python main.py --data_style style_color_chair_16 --data_content content_chair_train --data_dir ./data/03001627/ --alpha 0.3 --beta 10.0 --input_size 32 --output_size 256 --train --train_geo --gpu 0 --epoch 20
python main.py --data_style style_color_chair_16 --data_content content_chair_train --data_dir ./data/03001627/ --alpha 0.2 --beta 10.0 --input_size 32 --output_size 256 --train --train_tex --gpu 0 --epoch 20
python main.py --data_style style_color_building_8 --data_content content_building_train --data_dir ./data/00000000/ --alpha 0.5 --beta 10.0 --input_size 32 --output_size 256 --train --train_geo --asymmetry --gpu 0 --epoch 20
python main.py --data_style style_color_building_8 --data_content content_building_train --data_dir ./data/00000000/ --alpha 0.2 --beta 10.0 --input_size 32 --output_size 256 --train --train_tex --asymmetry --gpu 0 --epoch 20
These are examples for testing a model trained with 16 detailed cars. For other categories, please change the commands accordingly.
To output a few detailization results:
python main.py --data_style style_color_car_16 --data_content content_car_test --data_dir ./data/02958343/ --input_size 64 --output_size 512 --test --test_tex --gpu 0
Please add --asymmetry if testing the building category since we train the building category without using the symmetry assumption.
The output mesh can be found in folder samples or you can specify --sample_dir.
(Borrowed from DECOR-GAN) To test Strict-IOU, Loose-IOU, LP-IOU, Div-IOU, LP-F-score, Div-F-score:
python main.py --data_style style_color_chair_16 --data_content content_chair_test --data_dir ./data/03001627/ --input_size 32 --output_size 256 --prepvoxstyle --gpu 0
python main.py --data_style style_color_chair_16 --data_content content_chair_test --data_dir ./data/03001627/ --input_size 32 --output_size 256 --prepvox --gpu 0
python main.py --data_style style_color_chair_16 --data_content content_chair_test --data_dir ./data/03001627/ --input_size 32 --output_size 256 --evalvox --gpu 0
The first command prepares the patches in 16 detailed training shapes, thus --data_style is style_color_chair_16. Specifically, it removes duplicated patches in each detailed training shape and only keeps unique patches for faster computation in the following testing procedure. The unique patches are written to the folder unique_patches. Note that if you are testing multiple models, you do not have to run the first command every time -- just copy the folder unique_patches or make a symbolic link.
The second command runs the model and outputs the detailization results, in folder output_for_eval.
The third command evaluates the outputs. The results are written to folder eval_output ( result_IOU_mean.txt, result_LP_Div_Fscore_mean.txt, result_LP_Div_IOU_mean.txt ).
(Borrowed from DECOR-GAN) To test Cls-score:
python main.py --data_style style_color_chair_16 --data_content content_chair_all --data_dir ./data/03001627/ --input_size 32 --output_size 256 --prepimgreal --gpu 0
python main.py --data_style style_color_chair_16 --data_content content_chair_test --data_dir ./data/03001627/ --input_size 32 --output_size 256 --prepimg --gpu 0
python main.py --data_style style_color_chair_16 --data_content content_chair_all --data_dir ./data/03001627/ --input_size 32 --output_size 256 --evalimg --gpu 0
The first command prepares rendered views of all content shapes, thus --data_content is content_chair_all. The rendered views are written to the folder render_real_for_eval. Note that if you are testing multiple models, you do not have to run the first command every time -- just copy the folder render_real_for_eval or make a symbolic link.
The second command runs the model and outputs rendered views of the detailization results, in folder render_fake_for_eval.
The third command evaluates the outputs. The results are written to folder eval_output ( result_Cls_score.txt ).
- Build Cython module:
cd gui
python setup.py build_ext --inplace
- Make sure you put the checkpoint.pth in the
checkpointfolder, checkpoint can be found here - Change the
cpk_pathin thegui_demo.py - Run the GUI
python gui_demo.py --category 00000000
- Some basic modeling operations of GUI
add voxel - ctrl + left click
delete voxel - shift + left click
rotate - left click + drag
zoom in/out - scroll wheel
GUI currently only supports editing voxel from scratch, more input formats will be supported in the future.
https://github.com/qiminchen/ShaDDR/blob/main/gui/shaddr_demo.mp4