This is a forked Windows Installation Tutorial and the main codes will not be updated This forked GitHub project is intented for folks who have little to know command-line knowledge and want to install, train, and view 2D Gaussian Splats and derivative meshes. If you have used Nerftudio, 3DGS, or other similar command-line based radiance field projects, most likely you have already installed some or all of the depedencies required for this project.
Now let's get 3D Gaussian Splatting!
The section below is from the original GitHub page. Jump down to Overview to get started.
Project page | Paper | Video | Surfel Rasterizer (CUDA) | Surfel Rasterizer (Python) | DTU+COLMAP (3.5GB) | SIBR Viewer Pre-built for Windows
This repo contains the official implementation for the paper "2D Gaussian Splatting for Geometrically Accurate Radiance Fields". Our work represents a scene with a set of 2D oriented disks (surface elements) and rasterizes the surfels with perspective correct differentiable raseterization. Our work also develops regularizations that enhance the reconstruction quality. We also devise meshing approaches for Gaussian splatting.
2D Gaussian Splatting for Geometrically Accurate Radiance Fields is a project that is built upon the foundational work originally authored by Inria, Université Côte d'Azur, and MPI Informatik. The 2DGS workflow follows the same process as any other 3D gaussian splatting project with the additional layer of 3D Mesh extraction.
The following instructions will walk you through installation and all 4 main components of the 2DGS workflow: reparing data, training, mesh creation, and viewing, . If you get stuck or have questions, please check the original project's disucssion page first prior to asking questions here. Most likely someone else has already ran into your issue and has documented how to resolve it!
Despite my best efforts to compile the original environment provided by the researchers - I ended up just using the environment I set up for the original 3DGS project. The plus side, you will be able to run both projects at the end of this tutorial!
If you have already set up the original gaussian splatting environment, skip down to the 2DGS Installation instructions.
An NVIDIA 2XXX GPU or better. Preferably an RTX 3XXX or 4XXXX GPU with 12GB VRAM or more. Less VRAM means you will need to use less images, train less iterations, or downsample input images. This will be covered further down in the instructions.
This is the sofware dependencies you will need installed prior to installing the project. Many of these dependencies are shared with other NeRF projects.
- Git - You will need this to pull the code from GitHub. You can download it here . Follow default installation instructions. You can test to see if you have it already installed by typing
git --versioninto command prompt - Conda - I recommend using Anaconda because it's easy to install and manage environments in the future. MiniConda is a great lightweight alternative.
- CUDA Toolkit - this was tested with 11.8. Ensure you are not running 11.6 or 12+. You can download CUDA Toolkit here You can check which version of CUDA Toolkit you have installed by typing
nvcc --versioninto command prompt. - Visual Studio 2019 or newer - I found that the latest version of VS2022 will throw errors at you. I suggest downloading the latest Professional version of VS2019 here and then restarting your machine. You don't need to register it. note: make sure you add Desktop Development with C++ when installing
- COLMAP - Use the Windows binary, it's easy! You can download it here
- ImageMagik - This is for preparing your images. Download it here
- FFMPEG - Use this to extract images from video. Download it here
You will need to clone Inria's 3DGS repository from GitHub to set up the conda environment that you will run 2DGS (I know, confusing but trust me). Follow these steps to clone the repository:
- Open Windows Command Prompt by tying "cmd" into your search bar.
- Copy the below code into command prompt and press enter
git clone https://github.com/graphdeco-inria/gaussian-splatting --recursive
The folder will download to the root of our command line prompt with the name "Gaussian-Splatting". Typically in your C:User/<username> folder. For example, on my PC the folder is now located at C:User/Jonat/Guassian-Splatting
Next, you will need to create a Conda environment from the 3DGS code. Open command prompt and enter these lines below one at a time. The second line will compile the code which can take 10 minutes or longer. The last line will "activate" the conda environment. You will need to enter conda activate gaussian_splatting any time you want to run 2DGS (or 3DGS if you decide to use this project too!).
cd gaussian-splatting
SET DISTUTILS_USE_SDK=1
conda env create --file environment.yml
conda activate gaussian_splattingPlease note that this process assumes that you have CUDA SDK 11 installed.
After setting up the 3DGS environment, I suggest closing your command prompt window and start a fresh one. This will avoid cloning 2DGS code into the right wont.
git clone https://github.com/hbb1/2d-gaussian-splatting.git --recursive
Next, change directories into the newly cloned 2DGS folder:
cd 2d-gaussian-splatting
Activate the Gaussian Splatting conda environment that you set up earlier
conda activate gaussian_splatting
Now, install the 2DGS surfel rasterizer:
pip install submodules/diff-surfel-rasterization
Congrats! You have successfully installed 2DGS!!!
Our COLMAP loaders expect the following dataset structure in the source path location:
<location>
|---images
| |---<image 0>
| |---<image 1>
| |---...
|---sparse
|---0
|---cameras.bin
|---images.bin
|---points3D.bin
For rasterization, the camera models must be either a SIMPLE_PINHOLE or PINHOLE camera. We provide a converter script convert.py, to extract undistorted images and SfM information from input images. Optionally, you can use ImageMagick to resize the undistorted images. This rescaling is similar to MipNeRF360, i.e., it creates images with 1/2, 1/4 and 1/8 the original resolution in corresponding folders. To use them, please first install a recent version of COLMAP (ideally CUDA-powered) and ImageMagick. Put the images you want to use in a directory <location>/input.
<location>
|---input
|---<image 0>
|---<image 1>
|---...
If you have COLMAP and ImageMagick on your system path, you can simply run
python convert.py -s <location> [--resize] #If not resizing, ImageMagick is not neededAlternatively, you can use the optional parameters --colmap_executable and --magick_executable to point to the respective paths. Please note that on Windows, the executable should point to the COLMAP .bat file that takes care of setting the execution environment. Once done, <location> will contain the expected COLMAP data set structure with undistorted, resized input images, in addition to your original images and some temporary (distorted) data in the directory distorted.
If you have your own COLMAP dataset without undistortion (e.g., using OPENCV camera), you can try to just run the last part of the script: Put the images in input and the COLMAP info in a subdirectory distorted:
<location>
|---input
| |---<image 0>
| |---<image 1>
| |---...
|---distorted
|---database.db
|---sparse
|---0
|---...
Then run
python convert.py -s <location> --skip_matching [--resize] #If not resizing, ImageMagick is not neededCommand Line Arguments for convert.py
Flag to avoid using GPU in COLMAP.
Flag to indicate that COLMAP info is available for images.
Location of the inputs.
Which camera model to use for the early matching steps, OPENCV by default.
Flag for creating resized versions of input images.
Path to the COLMAP executable (.bat on Windows).
Path to the ImageMagick executable.
To train a scene, simply use
python train.py -s <path to COLMAP or NeRF Synthetic dataset>Commandline arguments for regularizations
--lambda_normal # hyperparameter for normal consistency
--lambda_distortion # hyperparameter for depth distortion
--depth_ratio # 0 for mean depth and 1 for median depth, 0 works for most casesTips for adjusting the parameters on your own dataset:
- For unbounded/large scenes, we suggest using mean depth, i.e.,
depth_ratio=0, for less "disk-aliasing" artifacts.
To export a mesh within a bounded volume, simply use
python render.py -m <path to pre-trained model> -s <path to COLMAP dataset> Commandline arguments you should adjust accordingly for meshing for bounded TSDF fusion, use
--depth_ratio # 0 for mean depth and 1 for median depth
--voxel_size # voxel size
--depth_trunc # depth truncationIf these arguments are not specified, the script will automatically estimate them using the camera information.
To export a mesh with an arbitrary size, we devised an unbounded TSDF fusion with space contraction and adaptive truncation.
python render.py -m <path to pre-trained model> -s <path to COLMAP dataset> --mesh_res 1024Assuming you have downloaded MipNeRF360, simply use
python train.py -s <path to m360>/<garden> -m output/m360/garden
# use our unbounded mesh extraction!!
python render.py -s <path to m360>/<garden> -m output/m360/garden --unbounded --skip_test --skip_train --mesh_res 1024
# or use the bounded mesh extraction if you focus on foreground
python render.py -s <path to m360>/<garden> -m output/m360/garden --skip_test --skip_train --mesh_res 1024If you have downloaded the DTU dataset, you can use
python train.py -s <path to dtu>/<scan105> -m output/date/scan105 -r 2 --depth_ratio 1
python render.py -r 2 --depth_ratio 1 --skip_test --skip_trainCustom Dataset: We use the same COLMAP loader as 3DGS, you can prepare your data following here.
For novel view synthesis on MipNeRF360 (which also works for other colmap datasets), use
python scripts/mipnerf_eval.py -m60 <path to the MipNeRF360 dataset>We provide Evaluation Results (Pretrained, Images).
Table Results
2DGS.Viewer.mp4
The Pre-built Viewer for Windows can be found here.
Firstly open the viewer,
<path to downloaded/compiled viewer>/bin/SIBR_remoteGaussian_app_rwdiand then
# Monitor the training process
python train.py -s <path to COLMAP or NeRF Synthetic dataset>
# View the trained model
python view.py -s <path to COLMAP or NeRF Synthetic dataset> -m <path to trained model> -
Training does not converge. If your camera's principal point does not lie at the image center, you may experience convergence issues. Our code only supports the ideal pinhole camera format, so you may need to make some modifications. Please follow the instructions provided here to make the necessary changes. We have also modified the rasterizer in the latest commit to support data accepted by 3DGS. To avoid further issues, please update to the latest commit.
-
No mesh / Broken mesh. When using the Bounded mesh extraction mode, it is necessary to adjust the
depth_truncparameter to perform TSDF fusion to extract meshes. On the other hand, Unbounded mesh extraction does not require tuning the parameters but is less efficient. -
Can 3DGS's viewer be used to visualize 2DGS? Technically, you can export 2DGS to 3DGS's ply file by appending an additional zero scale. However, due to the inaccurate affine projection of 3DGS's viewer, you may see some distorted artefacts. We are currently working on a viewer for 2DGS, so stay tuned for updates.
This project is built upon 3DGS. The TSDF fusion for extracting mesh is based on Open3D. The rendering script for MipNeRF360 is adopted from Multinerf, while the evaluation scripts for DTU and Tanks and Temples dataset are taken from DTUeval-python and TanksAndTemples, respectively. The fusing operation for accelerating the renderer is inspired by Han's repodcue. We thank all the authors for their great repos.
If you find our code or paper helps, please consider citing:
@inproceedings{Huang2DGS2024,
title={2D Gaussian Splatting for Geometrically Accurate Radiance Fields},
author={Huang, Binbin and Yu, Zehao and Chen, Anpei and Geiger, Andreas and Gao, Shenghua},
publisher = {Association for Computing Machinery},
booktitle = {SIGGRAPH 2024 Conference Papers},
year = {2024},
doi = {10.1145/3641519.3657428}
}