Finetune with lidar GT

[zhoujiawei3]

I would like to fine-tune the current model on an autonomous driving dataset to obtain more accurate depth estimation. May I directly use the sparse LiDAR image as the depth image to fine-tune the model?

[gbenga007]

@zhoujiawei3 , the second paper (MoGE-V2) shows a method for upsampling noisy or sparse depth maps using a 2 step process which involves removing troublesome points and then filling holes using a poisson completion method. They show very compelling qualitative results on LiDAR depth samples. Unfortunately though, the code for that poisson method is not released.

[zhoujiawei3]

@gbenga007 Hello, thank you for your reply. Since I am not very familiar with depth-related details and the refinement pipeline code has not been released yet, I find it difficult to fully reproduce the two-stage refinement process on my own. Therefore, I am wondering whether it might be feasible to fine-tune Moge-2 on an unseen dataset using sparse LiDAR ground truth in a simpler way.

My current idea is to introduce an additional mask during training to indicate the positions of the available LiDAR GT points. Then, using the existing inference pipeline to produce the depth map, I would compute the loss only at the masked (GT-provided) locations to fine-tune Moge-2. Do you have any suggestions or thoughts on this approach?

[gbenga007]

Yes, I think that is a good direction. I am afraid that if only a few points are used in supervision, you might run into issues with smoothness across the point map. But that might be the only option you have for now, definitely worth a shot!

All the best! Feel free to update on what you learn/find.

[abrahamezzeddine]

How about using Prompt Depth Anything to fill the sparse depth maps with additional lidar data?

https://promptda.github.io/