Complete list of training datasets

[kabouzeid]

Hi, congrats on this awesome work!

I noticed that the paper mentions "including" when listing the training datasets. Could you confirm whether the list is exhaustive? In particular, was the model trained on NYUv2? I ask because I get very good monocular depth estimation scores for VGGT on NYUv2 in my pipeline.

[jytime]

Hi @kabouzeid ,

Thanks for your trial. I think the publicly released checkpoint only used:

Co3Dv2 
BlendMVS 
DL3DV 
MegaDepth
Kubric 
WildRGB 
ScanNet
HyperSim 
Mapillary
Taskonomy
Habitat 
Replica 
MVS-Synth 
Virtual KITTI
Aria Synthetic Environments
Aria Digital Twin
A synthetic dataset of artist-created assets similar to Objaverse 

(Note that PointOdyssey is removed since we find dynamic scenes may not be really helpful. )

I will double check this again when I am back to laptop and if necessary revising here, but it should not happen. Especially, I can confirm NYU v2 was never trained or seen by our public checkpoint.

We also notice that our model can achieve really good-looking depth maps on monocular images, but did not check the quantitive results by now (sry I am not very familiar with the benchmarking protocol there). If possible, would you happy to share your benchmarking results here? For example, how good is our method compared to DepthAnything v2 in your test? Glad to share it with the community :)

[kabouzeid]

Thanks for the info! Here is a SotA table on the standard datasets. You're doing very well, especially given that you did not explicitly train for this and eval of all methods is only affine-invariant.

Note that e.g. MoGe uses a higher res than VGGT on most of these benchmarks, since it supports up to 700x700. For VGGT we use your image preprocessing which resizes to max 518. Especially for KITTI this results in very small image size: 518x154.

[kabouzeid]

Also here in this single-view experiment Depth+Cam tends to be ever so slightly stronger than Pointmap pred.

[jytime]

@kabouzeid Thanks a lot and glad to hear this! I’ve linked your GitHub account and this issue in the Readme with the recent commit. If you have an official GitHub repository or paper reference for this SotA table, please let me know, and I’ll update the link accordingly.

The observation about Depth+Cam is consistent with my findings in general-case testing, where Depth+Cam consistently outperforms Pointmap.

By the way, although only trained with a maximum size of 518, our model can indeed handle higher-resolution inputs (as discussed here). If it’s convenient, could you also evaluate our results at a higher input resolution?

Thanks again!

[kabouzeid]

I just extended the table of a previous paper, so there is nothing else to link to, thanks :)

My student @GonzaloMartinGarcia ran the inference pipeline for this, I'm sure we can get results for higher-res inputs soon, I'm also very interested in those.

[jytime]

As suggested by @Nik-V9, in case it’s helpful to others:

We used the Mapillary Metropolis dataset, instead of MPSD.

Mapillary Metropolis link: https://www.mapillary.com/dataset/metropolis

[insomniaaac]

Can you tell me how you handle the Habitat dataset? I noticed that the Habitat dataset consists of some glb files. Maybe I need to render these glb files?

[jytime]

HI @insomniaaac ,

You can use omnidata repo, https://github.com/EPFL-VILAB/omnidata/tree/f69ff3aedf983cb34c490a0afd0d29fdd83f6a1c, where Habitat has been rendered there.

[caifazhou]

has anyone tried to fine-tuning vggt using monocular depth dataset for further improving its performance?

[caifazhou]

I just extended the table of a previous paper, so there is nothing else to link to, thanks :)

My student @GonzaloMartinGarcia ran the inference pipeline for this, I'm sure we can get results for higher-res inputs soon, I'm also very interested in those.

Looking forward to this result!

[kabouzeid]

We did evaluate it back then, but I never got around to tidying up the results in a table. Bottom line was that it did not improve with higher resolution; on the contrary, it got slightly worse at higher res.

I later found out one needs to apply this fix, which we didn't do, and then it might become better at higher resolution: #32 (comment)

[wswdx]

HI @insomniaaac ,

You can use omnidata repo, https://github.com/EPFL-VILAB/omnidata/tree/f69ff3aedf983cb34c490a0afd0d29fdd83f6a1c, where Habitat has been rendered there.

Hi! Seems that the images from omnidata have no order (clustered by independent points and views), how did you sort the images to sequential frames for vggt data sampling?

[mmahdavian]

@jytime Is it possible for you provide the preprocessing and dataloader code for Mapillary Metropolis dataset? Or at least a guide for how to use the dataset for the vggt training.
Thank You