Added forward pass for depth rendering

[JonathonLuiten]

Adds CUDA code to also render the depth map (as well as the color map). Note this is ONLY the forward pass for this, feeding gradients back through this depth map won't work.

Note, it is unclear if this will result in slower rendering (my guess is that it shouldn't), but someone should check this probably before merging (e.g. train on garden with and without these changes), and if it is slower, then it should be added with a flag and off by default.

Also note that this by itself with break the main gaussian_splatting repo, so there needs to be the corresponding PR there merged too simultaneously. I will make a PR for that there too now.

[JonathonLuiten]

The required changes to the main repo can be found in the following PR:
graphdeco-inria/gaussian-splatting#20

[JonathonLuiten]

Hey @grgkopanas and @Snosixtyboo

I ran the speed test on this (tanks and temples / truck for 7000 iterations)

It results in 0 slowdown. I think it's safe to merge this.

Results:

WITH CHANGE:

Training progress: 23%|██▎ | 7000/30000 [02:50<12:04, 31.74it/s, Loss=0.0588340]
[ITER 7000] Evaluating train: L1 0.033260490745306015 PSNR 24.68514633178711 [13/07 15:25:22]

ORIGINAL:

Training progress: 23%|██▎ | 7000/30000 [02:50<12:07, 31.60it/s, Loss=0.0594036]
[ITER 7000] Evaluating train: L1 0.03345547989010811 PSNR 24.60102653503418 [13/07 15:32:42]

[BennetLeff]

feeding gradients back through this depth map won't work

Could you say why? Thanks!

[JonathonLuiten]

Because the backward pass for the depth hasn't been implemented. Unlike pytorch, the backward pass isn't automatically calculated in CUDA, and has to be written by hand.

[BennetLeff]

Ahh, by the way you had written it, I thought there was some more grand technical fault. Thanks!

[ray8828]

I always get memory error of this version with depth, seems there is hidden memory bug in this depth version

CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

[nishadgothoskar]

I'm curious whether a backward pass has been implemented for this

[jb-ye]

should it be D / (1 - T) here?

[junseo013]

@jb-ye I think so, too.

[zcc00210]

Adds CUDA code to also render the depth map (as well as the color map). Note this is ONLY the forward pass for this, feeding gradients back through this depth map won't work.

Note, it is unclear if this will result in slower rendering (my guess is that it shouldn't), but someone should check this probably before merging (e.g. train on garden with and without these changes), and if it is slower, then it should be added with a flag and off by default.

Also note that this by itself with break the main gaussian_splatting repo, so there needs to be the corresponding PR there merged too simultaneously. I will make a PR for that there too now.

Hi,
May I ask why after I use your depth visualization cuda code under this branch, each pixel of the rendered depth map output is 255,255,255?
Your prompt reply will be highly appreciated!@JonathonLuiten