diff --git a/docs/index.md b/docs/index.md index 2f3093834a..e516386f44 100644 --- a/docs/index.md +++ b/docs/index.md @@ -140,7 +140,7 @@ This documentation is organized into 3 parts: - [NeRF](nerfology/methods/nerf.md): OG Neural Radiance Fields - [Mip-NeRF](nerfology/methods/mipnerf.md): A Multiscale Representation for Anti-Aliasing Neural Radiance Fields - [TensoRF](nerfology/methods/tensorf.md): Tensorial Radiance Fields -- [Gaussian Splatting](nerfology/methods/splat.md): 3D Gaussian Splatting +- [Splatfacto](nerfology/methods/splat.md): Nerfstudio's Gaussian Splatting implementation (third_party_methods)= diff --git a/docs/nerfology/methods/index.md b/docs/nerfology/methods/index.md index ab15c1a188..6a7b296abf 100644 --- a/docs/nerfology/methods/index.md +++ b/docs/nerfology/methods/index.md @@ -27,7 +27,7 @@ The following methods are supported in nerfstudio: ```{toctree} :maxdepth: 1 Instant-NGP - 3D Gaussian Splatting + Splatfacto Instruct-NeRF2NeRF K-Planes LERF diff --git a/docs/nerfology/methods/splat.md b/docs/nerfology/methods/splat.md index 8e613e251b..8cfbefa44d 100644 --- a/docs/nerfology/methods/splat.md +++ b/docs/nerfology/methods/splat.md @@ -1,6 +1,6 @@ -# Gaussian Splatting -

Real-Time Radiance Field Rendering

- +# Splatfacto +

Nerfstudio's Gaussian Splatting Implementation

+ ```{button-link} https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/ :color: primary @@ -10,6 +10,8 @@ Paper Website [3D Gaussian Splatting](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/) was proposed in SIGGRAPH 2023 from INRIA, and is a completely different method of representing radiance fields by explicitly storing a collection of 3D volumetric gaussians. These can be "splatted", or projected, onto a 2D image provided a camera pose, and rasterized to obtain per-pixel colors. Because rasterization is very fast on GPUs, this method can render much faster than neural representations of radiance fields. +To avoid confusion with the original paper, we refer to nerfstudio's implementation as "Splatfacto", which will drift away from the original as more features are added. Just as Nerfacto is a blend of various different methods, Splatfacto will be a blend of different gaussian splatting methodologies. + ### Installation ```{button-link} https://docs.gsplat.studio/ @@ -18,22 +20,22 @@ Paper Website GSplat ``` -Nerfstudio uses [gsplat](https://github.com/nerfstudio-project/gsplat) as its gaussian rasterization backend, an in-house re-implementation which is designed to be more developer friendly. This can be installed with `pip install gsplat`. The associated CUDA code will be compiled the first time gaussian splatting is executed. Some users with PyTorch 2.0 have experienced issues with this, which can be resolved by either installing gsplat from source, or upgrading torch to 2.1. +Nerfstudio uses [gsplat](https://github.com/nerfstudio-project/gsplat) as its gaussian rasterization backend, an in-house re-implementation which is designed to be more developer friendly. This can be installed with `pip install gsplat`. The associated CUDA code will be compiled the first time gsplat is executed. Some users with PyTorch 2.0 have experienced issues with this, which can be resolved by either installing gsplat from source, or upgrading torch to 2.1. ### Data -Gaussian Splatting works much better if you initialize it from pre-existing geometry, such as SfM points from COLMAP. COLMAP datasets or datasets from `ns-process-data` will automatically save these points and initialize gaussians on them. Other datasets currently do not support initialization, and will initialize gaussians randomly. Initializing from other data inputs (i.e. depth from phone app scanners) may be supported in the future. +Gaussian spltting works much better if you initialize it from pre-existing geometry, such as SfM points from COLMAP. COLMAP datasets or datasets from `ns-process-data` will automatically save these points and initialize gaussians on them. Other datasets currently do not support initialization, and will initialize gaussians randomly. Initializing from other data inputs (i.e. depth from phone app scanners) may be supported in the future. -Because gaussian splatting trains on *full images* instead of bundles of rays, there is a new datamanager in `full_images_datamanager.py` which undistorts input images, caches them, and provides single images at each train step. +Because the method trains on *full images* instead of bundles of rays, there is a new datamanager in `full_images_datamanager.py` which undistorts input images, caches them, and provides single images at each train step. ### Running the Method -To run gaussian splatting, run `ns-train gaussian-splatting --data `. Just like NeRF methods, the splat can be interactively viewed in the web-viewer, loaded from a checkpoint, rendered, and exported. +To run splatfacto, run `ns-train splatfacto --data `. Just like NeRF methods, the splat can be interactively viewed in the web-viewer, loaded from a checkpoint, rendered, and exported. #### Quality and Regularization The default settings provided maintain a balance between speed, quality, and splat file size, but if you care more about quality than training speed or size, you can decrease the alpha cull threshold -(threshold to delete translucent gaussians) and disable culling after 15k steps like so: `ns-train gaussian-splatting --pipeline.model.cull_scale_thresh=0.005 --pipeline.model.continue_cull_post_densification=False --data ` +(threshold to delete translucent gaussians) and disable culling after 15k steps like so: `ns-train splatfacto --pipeline.model.cull_scale_thresh=0.005 --pipeline.model.continue_cull_post_densification=False --data ` -A common artifact in splatting is long, spikey gaussians. [PhysGaussian](https://xpandora.github.io/PhysGaussian/) proposes an example of a scale-regularizer that encourages gaussians to be more evenly shaped. To enable this, set the `use_scale_regularization` flag to `True`. +A common artifact in splatting is long, spikey gaussians. [PhysGaussian](https://xpandora.github.io/PhysGaussian/) proposes a scale regularizer that encourages gaussians to be more evenly shaped. To enable this, set the `use_scale_regularization` flag to `True`. ### Details For more details on the method, see the [original paper](https://arxiv.org/abs/2308.04079). Additionally, for a detailed derivation of the gradients used in the gsplat library, see [here](https://arxiv.org/abs/2312.02121). @@ -41,7 +43,7 @@ For more details on the method, see the [original paper](https://arxiv.org/abs/2 ### Exporting splats Gaussian splats can be exported as a `.ply` file which are ingestable by a variety of online web viewers. You can do this via the viewer, or `ns-export gaussian-splat --load-config --output-dir exports/splat`. Currently splats can only be exported from trained splats, not from nerfacto. -Nerfstudio gaussian splat exports currently supports multiple third-party splat viewers: +Nerfstudio's splat export currently supports multiple third-party splat viewers: - [Polycam Viewer](https://poly.cam/tools/gaussian-splatting) - [Playcanvas SuperSplat](https://playcanvas.com/super-splat) - [WebGL Viewer by antimatter15](https://antimatter15.com/splat/) @@ -50,6 +52,8 @@ Nerfstudio gaussian splat exports currently supports multiple third-party splat ### FAQ - Can I export a mesh or pointcloud? -Currently these export options are not supported, but may become in the future. Contributions are always welcome! + +Currently these export options are not supported, but may be in the future. Contributions are always welcome! - Can I render fisheye, equirectangular, orthographic images? -Currently, no. Gaussian splatting assumes a perspective camera for its rasterization pipeline. Implementing other camera models is of interest but not currently planned. + +Currently, no. Gaussian rasterization assumes a perspective camera for its rasterization pipeline. Implementing other camera models is of interest but not currently planned. diff --git a/nerfstudio/configs/method_configs.py b/nerfstudio/configs/method_configs.py index f60450163b..dd842e8ab6 100644 --- a/nerfstudio/configs/method_configs.py +++ b/nerfstudio/configs/method_configs.py @@ -51,7 +51,6 @@ from nerfstudio.field_components.temporal_distortions import TemporalDistortionKind from nerfstudio.fields.sdf_field import SDFFieldConfig from nerfstudio.models.depth_nerfacto import DepthNerfactoModelConfig -from nerfstudio.models.gaussian_splatting import GaussianSplattingModelConfig from nerfstudio.models.generfacto import GenerfactoModelConfig from nerfstudio.models.instant_ngp import InstantNGPModelConfig from nerfstudio.models.mipnerf import MipNerfModel @@ -59,6 +58,7 @@ from nerfstudio.models.neus import NeuSModelConfig from nerfstudio.models.neus_facto import NeuSFactoModelConfig from nerfstudio.models.semantic_nerfw import SemanticNerfWModelConfig +from nerfstudio.models.splatfacto import SplatfactoModelConfig from nerfstudio.models.tensorf import TensoRFModelConfig from nerfstudio.models.vanilla_nerf import NeRFModel, VanillaModelConfig from nerfstudio.pipelines.base_pipeline import VanillaPipelineConfig @@ -80,7 +80,7 @@ "generfacto": "Generative Text to NeRF model", "neus": "Implementation of NeuS. (slow)", "neus-facto": "Implementation of NeuS-Facto. (slow)", - "gaussian-splatting": "Gaussian Splatting model", + "splatfacto": "Gaussian Splatting model", } method_configs["nerfacto"] = TrainerConfig( @@ -588,8 +588,8 @@ vis="viewer", ) -method_configs["gaussian-splatting"] = TrainerConfig( - method_name="gaussian-splatting", +method_configs["splatfacto"] = TrainerConfig( + method_name="splatfacto", steps_per_eval_image=100, steps_per_eval_batch=0, steps_per_save=2000, @@ -601,7 +601,7 @@ datamanager=FullImageDatamanagerConfig( dataparser=NerfstudioDataParserConfig(load_3D_points=True), ), - model=GaussianSplattingModelConfig(), + model=SplatfactoModelConfig(), ), optimizers={ "xyz": { diff --git a/nerfstudio/data/dataparsers/nerfstudio_dataparser.py b/nerfstudio/data/dataparsers/nerfstudio_dataparser.py index 892ed0684b..aa06c53d33 100644 --- a/nerfstudio/data/dataparsers/nerfstudio_dataparser.py +++ b/nerfstudio/data/dataparsers/nerfstudio_dataparser.py @@ -391,7 +391,7 @@ def _generate_dataparser_outputs(self, split="train"): else: if not self.prompted_user: CONSOLE.print( - "[bold yellow]Warning: load_3D_points set to true but no point cloud found. gaussian-splatting models will use random point cloud initialization." + "[bold yellow]Warning: load_3D_points set to true but no point cloud found. splatfacto will use random point cloud initialization." ) ply_file_path = None diff --git a/nerfstudio/models/gaussian_splatting.py b/nerfstudio/models/splatfacto.py similarity index 98% rename from nerfstudio/models/gaussian_splatting.py rename to nerfstudio/models/splatfacto.py index 2365e43f7c..20e4b86f87 100644 --- a/nerfstudio/models/gaussian_splatting.py +++ b/nerfstudio/models/splatfacto.py @@ -99,10 +99,10 @@ def projection_matrix(znear, zfar, fovx, fovy, device: Union[str, torch.device] @dataclass -class GaussianSplattingModelConfig(ModelConfig): - """Gaussian Splatting Model Config""" +class SplatfactoModelConfig(ModelConfig): + """Splatfacto Model Config, nerfstudio's implementation of Gaussian Splatting""" - _target: Type = field(default_factory=lambda: GaussianSplattingModel) + _target: Type = field(default_factory=lambda: SplatfactoModel) warmup_length: int = 500 """period of steps where refinement is turned off""" refine_every: int = 100 @@ -149,14 +149,14 @@ class GaussianSplattingModelConfig(ModelConfig): """ -class GaussianSplattingModel(Model): - """Gaussian Splatting model +class SplatfactoModel(Model): + """Nerfstudio's implementation of Gaussian Splatting Args: - config: Gaussian Splatting configuration to instantiate model + config: Splatfacto configuration to instantiate model """ - config: GaussianSplattingModelConfig + config: SplatfactoModelConfig def __init__( self, diff --git a/nerfstudio/scripts/exporter.py b/nerfstudio/scripts/exporter.py index 45be2e84cc..e526268890 100644 --- a/nerfstudio/scripts/exporter.py +++ b/nerfstudio/scripts/exporter.py @@ -40,7 +40,7 @@ from nerfstudio.exporter.exporter_utils import collect_camera_poses, generate_point_cloud, get_mesh_from_filename from nerfstudio.exporter.marching_cubes import generate_mesh_with_multires_marching_cubes from nerfstudio.fields.sdf_field import SDFField # noqa -from nerfstudio.models.gaussian_splatting import GaussianSplattingModel +from nerfstudio.models.splatfacto import SplatfactoModel from nerfstudio.pipelines.base_pipeline import Pipeline, VanillaPipeline from nerfstudio.utils.eval_utils import eval_setup from nerfstudio.utils.rich_utils import CONSOLE @@ -488,9 +488,9 @@ def main(self) -> None: _, pipeline, _, _ = eval_setup(self.load_config) - assert isinstance(pipeline.model, GaussianSplattingModel) + assert isinstance(pipeline.model, SplatfactoModel) - model: GaussianSplattingModel = pipeline.model + model: SplatfactoModel = pipeline.model filename = self.output_dir / "splat.ply" diff --git a/nerfstudio/viewer/export_panel.py b/nerfstudio/viewer/export_panel.py index 5d93a20dd8..76808a2725 100644 --- a/nerfstudio/viewer/export_panel.py +++ b/nerfstudio/viewer/export_panel.py @@ -22,7 +22,7 @@ from nerfstudio.data.scene_box import OrientedBox from nerfstudio.models.base_model import Model -from nerfstudio.models.gaussian_splatting import GaussianSplattingModel +from nerfstudio.models.splatfacto import SplatfactoModel from nerfstudio.viewer.control_panel import ControlPanel @@ -32,7 +32,7 @@ def populate_export_tab( config_path: Path, viewer_model: Model, ) -> None: - viewing_gsplat = isinstance(viewer_model, GaussianSplattingModel) + viewing_gsplat = isinstance(viewer_model, SplatfactoModel) if not viewing_gsplat: crop_output = server.add_gui_checkbox("Use Crop", False) diff --git a/nerfstudio/viewer/render_state_machine.py b/nerfstudio/viewer/render_state_machine.py index 74fde38336..d116a190e4 100644 --- a/nerfstudio/viewer/render_state_machine.py +++ b/nerfstudio/viewer/render_state_machine.py @@ -28,7 +28,7 @@ from nerfstudio.cameras.cameras import Cameras from nerfstudio.model_components.renderers import background_color_override_context -from nerfstudio.models.gaussian_splatting import GaussianSplattingModel +from nerfstudio.models.splatfacto import SplatfactoModel from nerfstudio.utils import colormaps, writer from nerfstudio.utils.writer import GLOBAL_BUFFER, EventName, TimeWriter from nerfstudio.viewer.utils import CameraState, get_camera @@ -136,7 +136,7 @@ def _render_img(self, camera_state: CameraState): with TimeWriter(None, None, write=False) as vis_t: with self.viewer.train_lock if self.viewer.train_lock is not None else contextlib.nullcontext(): - if isinstance(self.viewer.get_model(), GaussianSplattingModel): + if isinstance(self.viewer.get_model(), SplatfactoModel): color = self.viewer.control_panel.background_color background_color = torch.tensor( [color[0] / 255.0, color[1] / 255.0, color[2] / 255.0], @@ -168,7 +168,7 @@ def _render_img(self, camera_state: CameraState): self.viewer.get_model().train() num_rays = (camera.height * camera.width).item() if self.viewer.control_panel.layer_depth: - if isinstance(self.viewer.get_model(), GaussianSplattingModel): + if isinstance(self.viewer.get_model(), SplatfactoModel): # Gaussians render much faster than we can send depth images, so we do some downsampling. assert len(outputs["depth"].shape) == 3 assert outputs["depth"].shape[-1] == 1 diff --git a/nerfstudio/viewer/viewer.py b/nerfstudio/viewer/viewer.py index 579dbb5cc4..4480bf214e 100644 --- a/nerfstudio/viewer/viewer.py +++ b/nerfstudio/viewer/viewer.py @@ -32,7 +32,7 @@ from nerfstudio.configs import base_config as cfg from nerfstudio.data.datasets.base_dataset import InputDataset from nerfstudio.models.base_model import Model -from nerfstudio.models.gaussian_splatting import GaussianSplattingModel +from nerfstudio.models.splatfacto import SplatfactoModel from nerfstudio.pipelines.base_pipeline import Pipeline from nerfstudio.utils.decorators import check_main_thread, decorate_all from nerfstudio.utils.writer import GLOBAL_BUFFER, EventName @@ -250,7 +250,7 @@ def nested_folder_install(folder_labels: List[str], prev_labels: List[str], elem # Diagnostics for Gaussian Splatting: where the points are at the start of training. # This is hidden by default, it can be shown from the Viser UI's scene tree table. - if isinstance(pipeline.model, GaussianSplattingModel): + if isinstance(pipeline.model, SplatfactoModel): self.viser_server.add_point_cloud( "/gaussian_splatting_initial_points", points=pipeline.model.means.numpy(force=True) * VISER_NERFSTUDIO_SCALE_RATIO, diff --git a/nerfstudio/viewer_legacy/server/render_state_machine.py b/nerfstudio/viewer_legacy/server/render_state_machine.py index d842598d1a..a3c0524906 100644 --- a/nerfstudio/viewer_legacy/server/render_state_machine.py +++ b/nerfstudio/viewer_legacy/server/render_state_machine.py @@ -24,7 +24,7 @@ from nerfstudio.cameras.cameras import Cameras from nerfstudio.model_components.renderers import background_color_override_context -from nerfstudio.models.gaussian_splatting import GaussianSplattingModel +from nerfstudio.models.splatfacto import SplatfactoModel from nerfstudio.utils import colormaps, writer from nerfstudio.utils.writer import GLOBAL_BUFFER, EventName, TimeWriter from nerfstudio.viewer_legacy.server import viewer_utils @@ -130,7 +130,7 @@ def _render_img(self, cam_msg: CameraMessage): with self.viewer.train_lock if self.viewer.train_lock is not None else contextlib.nullcontext(): # TODO jake-austin: Make this check whether the model inherits from a camera based model or a ray based model # TODO Zhuoyang: First made some dummy judgements, need to be fixed later - isGaussianSplattingModel = isinstance(self.viewer.get_model(), GaussianSplattingModel) + isGaussianSplattingModel = isinstance(self.viewer.get_model(), SplatfactoModel) if isGaussianSplattingModel: # TODO fix me before ship camera_ray_bundle = camera.generate_rays(camera_indices=0, aabb_box=self.viewer.get_model().render_aabb) diff --git a/pyproject.toml b/pyproject.toml index c35ea04788..ad276c0666 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -97,7 +97,9 @@ dev = [ "opencv-stubs==0.0.7", "transformers==4.29.2", "pyright==1.1.331", - "projectaria_tools[all]>=1.2.0", + # NOTE: Disabling projectaria-tools because it doesn't have prebuilt windows wheels + # Syntax comes from here: https://pip.pypa.io/en/stable/reference/requirement-specifiers/ + "projectaria-tools>=1.3.1; sys_platform != 'win32'", ] # Documentation related packages diff --git a/tests/test_train.py b/tests/test_train.py index 45d8e348f0..82e198373b 100644 --- a/tests/test_train.py +++ b/tests/test_train.py @@ -26,7 +26,7 @@ "neus", "generfacto", "neus-facto", - "gaussian-splatting", + "splatfacto", ]