Gaussian splat spherical harmonics support

[keyboardspecialist]

Description

Adds spherical harmonic support for Gaussian splats in the SPZ format. Supports degrees 1-3 and auto detects what is available. This brings full SPZ support to CesiumJS.

Issue number and link

Testing plan

View the same asset tiled with 3 degrees of SH in the current main build versus the CI brand build:

No SH Support
SH Support

Author checklist

• I have submitted a Contributor License Agreement
• I have added my name to CONTRIBUTORS.md
• I have updated CHANGES.md with a short summary of my change
• I have added or updated unit tests to ensure consistent code coverage
• I have updated the inline documentation, and included code examples where relevant
• I have performed a self-review of my code

[github-actions]

Thank you for the pull request, @keyboardspecialist!

✅ We can confirm we have a CLA on file for you.

[javagl]

What is the SH degree of that example asset?

In any case, I think that proper SH support can hardly be implemented (and certainly not tested properly) based on the current SPZ loader, because it does not properly decode the SH data - unless the CesiumJS code is anticipating that wrong encoding, and compensating manually, in which case we just have to hope that it will never be fixed in the SPZ loader.

[keyboardspecialist]

The underlying SPZ wasm module decodes it. The raw decoded SH buffer is all we need.

Example is degree 3

[javagl]

Asked for clarification at drumath2237/spz-loader#36 (comment)

There is an effect that is clearly visible:

Differences to BabylonJS may be due to coordinate system differences (it's difficult)

Test data:

Shs Example.zip

[keyboardspecialist]

Babylon makes their lives simpler by rotating the world around the model.

The model's inverse rotation matrix is applied to the view angle which should give us the correct sampling.

One thing to keep in mind is we do apply a axis correction matrix which may contribute to what you are seeing. When I load our tiled SPZ into Babylon its sideways. That 90 degree rotation might be what you are seeing?

[javagl]

I think that the different coordinate system conventions can likely explain some differences.

Until now, I didn't dive into the maths behind the spherical harmonics. Yes, it's a bunch of coefficients that are mushed together with the view direction and eventually yield a "color". But I don't thoroughly understand the 'meaning of the values'. For example, I'd really like to create such a unit cube with spherical harmonics that looks

1. red from the right, and cyan from the left
2. green from the top and magenta from the bottom
3. blue from the front and orange from the back

This could help to see whether the SHs are taken into account correctly, and whether there are any orientation issues. (If someone knows, from the tip of the head, what the SH values would have to be for that, I'd create an example - until then, this is scheduled in the 'When I have way too much time on my hands'-section of my TODO list).

[jjhembd]

I'll put in a +1 for the "unit cube" test data. Without that, we don't have any way of knowing if the SH are rendered correctly, or wrongly but canceled by an opposite error in the data generation, or just wrong.

Here's what I see for the 2 Sandcastles.
No SH (before the PR):

With SH (after this PR):

At first glance it looks like a change in the scene lighting direction/intensity.

[javagl]

Creating predefined test data for this is a bit tricky, though. Deriving the right values from the shader code is close to impossible - there's a reason why there is some magic "iterative learner" blackbox generating these coefficients in "real" applications. I considered a lazy/sneaky approach, and just created a GLB of such a (solid) cube, recorded a video of that spinning it in all directions, and uploaded that to poly.cam, but ... that simply refuses to generate splats from that. (Even if it worked, it might not even possible to derive the right values from that).

A bit of trial and error gave the "red-cyan" effect, but only that, and only along the diagonal...

... so I think that it might be possible, and maybe even "easy" for someone who knows what he's doing. Not me, at least, not at the moment.

[ggetz]

Hi @keyboardspecialist, what's the status here? Any input on the testing data discussion above?

[jjhembd]

Where can we find a spec or description of the SPZ format? Is the format stable?
A link to the spec, perhaps in GltfVertexBufferLoader, could go a long way to shortening the spin-up time for future maintainers.

[javagl]

The thing that is closest to a spec is the README of https://github.com/nianticlabs/spz . The actual truth is in the code. Request for clarification are in nianticlabs/spz#42 . This does not yet cover the details of SHs and their layout, but maybe what's in the README is enough... (?)

[wongshek]

@jjhembd Here is a small ply data where the correctness of higher-order SH rendering can be clearly perceived. It would be great if it could help.

simple_board.ply.zip

Below is a comparison screenshot from Supersplat.

[javagl]

For example, I'd really like to create such a unit cube with spherical harmonics that looks

red from the right, and cyan from the left
green from the top and magenta from the bottom
blue from the front and orange from the back

This could help to see whether the SHs are taken into account correctly, and whether there are any orientation issues. (If someone knows, from the tip of the head, what the SH values would have to be for that, I'd create an example - until then, this is scheduled in the 'When I have way too much time on my hands'-section of my TODO list).

I guess that it's hard to just "come up" with the proper SH coefficients for that.

So I ported the GLSL code from the GaussianSplattingViewer to Java, took the view- and projection matrix and the camera positions for the top/bottom/left/right/front/back view configuration as the inputs, the desired color as the outputs, and interpreted the whole thing as an optimization problem that I fed into the Apache BOBYQAOptimizer.
Exactly what any sane person would do 🤡

The result as ASCII PLY, SPZ, GLB, and a matching tileset JSON and Sandcastle:

Splat SH orientation experiment 2025-08-13.zip

This is what it looks like:

• blue from the front, yellow from the back
• green from the top, magenta from the bottom
• red from the right, cyan from the left

(This is rendered with the viewer from JSplat. BabylonJS shows the same for SPZ. For PLY, there seems to be an orientation issue, or I'm writing the coefficients in the wrong order... who knows what's "right" or "wrong" here...)

I tried to check the different views in CesiumJS:

Looks about right. The bottom is missing - let's just assume it's magenta.