Implement new rasterization mode using opacity compensation factor

[jb-ye]

See previous threads here: nerfstudio-project/gsplat#117

The PR must be merged after this PR (version bump of gsplat lib, nerfstudio-project/gsplat#124)

[ichsan2895]

So we must add --pipeline.model.rasterize-mode 'standard' after ns-train splatfacto to see the new result? then "classic" mode is same as Original splatfacto?

[ichsan2895]

I tested on another dataset, so far only 4 tested (kitchen, truck, playroom, and desolation). Where 3 of them (truck, playroom, and desolation) I see a little bit improvement.

Kitchen dataset with splatfacto-v101:

Kitchen dataset with --pipeline.model.rasterize-mode 'standard':

playroom dataset:
Left = Using --pipeline.model.rasterize-mode 'standard'
PSNR = 31,44 dB

Right = using Splatfacto v1.0.1, so it don't use this rasterization.
PSNR = 31,38 dB

[ichsan2895]

truck dataset:
Left = Using --pipeline.model.rasterize-mode 'standard'
PSNR = 26,19 dB

Right = using Splatfacto v1.0.1, so it don't use this rasterization.
PSNR = 25,97 dB

[jb-ye]

Currently only nerfstudio viewer is supported to display the results correctly. Most other viewers which were designed for the classic mode must accommodate the same change to show the right the results of this new feature.

[ichsan2895]

Currently only nerfstudio viewer is supported to display the results correctly. Most other viewers which were designed for the classic mode must accommodate the same change to show the right the results of this new feature.

for truck and kitchen dataset that I have shown before, it was a frame from video which come from ns-render result. So, it does not screenshot the 3DGS Web viewer.

for playroom dataset, yes it is the screenshot from mkellogg 3D WebGL Gaussian Splat.

[jb-ye]

It is nice to see that this change also improves PSNR a bit on validation images

[kerrj]

sgtm!

[jb-ye]

The new mode name is called antialiased

[f-dy]

IMHO this should become the default, because:

• "antialiased" hold a closer-to-reality physical representation of the scene
• it doesn't hurt eval results when eval images are at the same distance as train images (there's even a slight improvement)
• it would certainly improve eval results for images that are closer or farther

Note that the best test data for this is to use something with a fine and regular texture (eg coarse-thread fabric, things with grids).

As we are getting closer to a physical representation, I hope that my proposed change on working in linear space rather than gamma-compressed (#925) will have a visible impact some day.

[f-dy]

The fact that third-party viewers don't render 2D Gaussians correctly is their own issue (some code don't even have the +0.3 trick)

[gonzalle]

Is it the same rasterization than this solution ???
it has a viewer based on mkkellog's one... (unfortunatly, unavailable in the repo)

[jb-ye]

@gonzalle It is not exactly the same implementation as mip-splatting, but is in similar spirit and simpler. I don't think you can use that viewer to display PLY exports from splatfacto.

[jinhwanlazy]

@jb-ye Could you please clarify why the opacity compensation is not being applied to the rasterization of the depth image?

nerfstudio/nerfstudio/models/splatfacto.py

Line 814 in 29ddd2c

torch.sigmoid(opacities_crop),

[jb-ye]

@jinhwanlazy Nice catch. I think you are right, this is a bug we have to fix. Could you send out a PR to fix it?