Is there a way of using RGBD images as the input?

[MaxPalmer-UH]

This solution extract depth data for photos as part of the pipeline (plus uncertainties?). Is there a way of providing RGBD data directly from depth cameras?

[barbararoessle]

Hi, sure, it is also possible to optimize NeRF with the depth loss and depth-guided sampling from RGBD data.
F.ex. in the code what we load as ground truth depth (gt_depths, gt_valid_depths here) is the sensor depth from ScanNet. You could use this depth data, instead of running depth completion on the sparse depth from SfM. When assigning the train depth maps: depths[i_train, :, :, 0] is the depth in meters and depths[i_train, :, :, 1] is the standard deviation in meters. The standard deviation can be set according to the accuracy of the depth sensor.
When using a different source of depth input you probably need to find a suitable depth loss weight.

[liang3588]

If I have the dense depth maps already, how to compute the standard deviation? Can it be calculated by the depth of the surrounding pixels?