Problems when 3 input views

[Gaozihui]

Thanks for your great work!

I have two problems about 3 input views:

1. I test 2 views and 3 views respectively with json files:
   {"41bcd011f99bfb66":{"context": [4, 50], "target": [7, 19, 49]},"5aca87f95a9412c6": {"context": [58, 133], "target": [84, 129]}, "322261824c4a3003": {"context": [33, 78], "target": [38, 61]}}
{"41bcd011f99bfb66":{"context": [4, 30, 50], "target": [7, 19, 49]},"5aca87f95a9412c6": {"context": [58, 102, 133], "target": [84, 129]}, "322261824c4a3003": {"context": [33, 60, 78], "target": [38, 61]}}
   but I find the 2 input views get better results than 3 views. The psnrs are 30.852 and 26.153 respectively. I believe that more input views should give better results.

2. How to exporting .ply File from 3 input views? I have succeed getting .ply from 2 views. Then I add dataset.view_sampler.num_context_views=3 and change the scenes to ("41bcd011f99bfb66", 4,30, 50, 6.0, [4,20,30,50], 1.4, 19) in generate_point_cloud_figure_mvsplat.py but it didn't work.

Thank you for your help in advance.

[donydchen]

Hi @Gaozihui, sorry for the late reply, I've been busy with several deadlines previously.

Regarding the 3-view performance, the model was trained on 2 views, so typically, it would work best when tested with the same number of views. More input views indeed bring more information but might also lead to unintentional noise. Therefore, for some specific scenes, it could be possible that more views might not improve the scores.

A better way might be to average over more scenes, which should give a more reliable conclusion. For example, we verify on the DTU dataset that 3-views indeed perform better than 2-views. Related discussions can be found in "Assessing more-view quality" in paper Sec. 4.2, and associated commands are at

mvsplat/more_commands.sh

Lines 45 to 61 in 6cf9dee

	# Table 2: RealEstate10K -> DTU (2 context views)
	python -m src.main +experiment=dtu \
	checkpointing.load=checkpoints/re10k.ckpt \
	mode=test \
	dataset/view_sampler=evaluation \
	dataset.view_sampler.index_path=assets/evaluation_index_dtu_nctx2.json \
	test.compute_scores=true
	# RealEstate10K -> DTU (3 context views)
	python -m src.main +experiment=dtu \
	checkpointing.load=checkpoints/re10k.ckpt \
	mode=test \
	dataset/view_sampler=evaluation \
	dataset.view_sampler.index_path=assets/evaluation_index_dtu_nctx3.json \
	dataset.view_sampler.num_context_views=3 \
	wandb.name=dtu/views3 \
	test.compute_scores=true

---

We only tested with 2 views when exporting point cloud. You might need to modify the script accordingly to make it work on 3 views, it should not be complicated. You can manually set the context view number at

mvsplat/src/paper/generate_point_cloud_figure_mvsplat.py

Lines 142 to 148 in 6cf9dee

	view_sampler_cfg = ViewSamplerArbitraryCfg(
	"arbitrary",
	2,
	2,
	context_views=list(context_indices),
	target_views=[0, 0], # use [40, 80] for teaser
	)

e.g., change it to

        view_sampler_cfg = ViewSamplerArbitraryCfg(
            "arbitrary",
            3,   # number of context views
            2,
            context_views=list(context_indices),
            target_views=[0, 0],  # use [40, 80] for teaser
        )