Differentiable Hybrid Renderer for 3D Gaussians & Mesh

Current Version: v0.1 (July 02, 2025)

TL;DR

• Combining Gaussians and meshes lets you leverage the strengths of both for more flexible and realistic 3D scene rendering.
• Gmesh supports differentiable rendering of mixed 3D Gaussians and meshes within a single scene.
• Built on gsplat and PyTorch3D, Gmesh is fully PyTorch-native.

Examples

Please see ./examples:

1. 3D Mesh Rendering via Pytorch3D: Link
2. 3DGS via gsplat: Link
3. Hybrid Gaussians and Mesh Rendering via Gmesh: Link

Setup

# create conda virtual environment
conda create --name gmesh -y python=3.10
conda activate gmesh

# install cuda toolkit
conda install -c "nvidia/label/cuda-11.7.1" cuda-toolkit ninja
conda install -c conda-forge cudatoolkit=11.7 cudatoolkit-dev=11.7

# install PyTorch
pip install torch==2.0.1 torchvision --index-url https://download.pytorch.org/whl/cu117

# install other dependencies
pip install -r requirements.txt

# install Gmesh
pip install -e .

Tip

I have only tested the following library versions: CUDA 11.7, PyTorch 2.0.1, PyTorch3D V0.7.8, and gsplat v1.5.2. If you encounter any issues or unexpected results, please double-check that you have installed the correct versions.

Design Details

Pipeline

Warning

Using depth maps for blending can only approximate the rendering; without a z-buffer, accurate blending is not possible.

Camera Coordinates

Demo

You can play with our interactive demonstration to see the rendering results:

python interactive_demo.py

Watch the demo video on YouTube

Limitations and Todos

The current version has the following main limitations:

• The depth map of 3DGS may not reflect the nearest depth.
• The rendered color of overlapping Gaussians may still be visible in the final image, even when those Gaussians are occluded by a mesh.
• 3D Gaussians' are not relightable.
• Does not support mesh opacity.
• Does not support mesh texture map.
• Does not support batch rendering.

Todos:

• Support z-buffering.
• Support mesh texture map.
• Support batch (batch of camera poses) rendering.
• Add two additional mesh to Gaussians conversion methods.
• Add some training examples.

Acknowledgement and Disclaimer

The code can be used for research purposes, provided that the terms of the licenses of any third-party libraries, data or dependencies are followed. For commercial use, the parts of code we wrote are for free, but please be aware to get permissions from any third-party to use their code or dependencies. We do not assume any responsibility for any issues, damages, or liabilities that may arise from the use of this code. Users are responsible for ensuring compliance with any legal requirements, including licensing terms and conditions, and for verifying that the code is suitable for their intended purposes.

Note

The example 3D mesh used in ./assets/data/spot.obj is sourced from: https://www.cs.cmu.edu/~kmcrane/Projects/ModelRepository/#spot. Please ensure you adhere to the license agreement provided on that site. We do not claim ownership of this mesh file.

🧸 Citation

Please cite this work if you find it useful in your projects.

@software{Yan_GMesh_Differentiable_Hybrid_2025,
    author = {Yan, Peizhi},
    month = jul,
    title = {{GMesh: Differentiable Hybrid Renderer for 3D Gaussians and Meshes}},
    url = {https://github.com/PeizhiYan/gmesh},
    version = {0.1},
    year = {2025}
}