About the result on Mapillary

[Jeff-Zilence]

Thank you for this nice code base and paper.
I am trying to reproduce the result on Mapillary following the training from scratch instruction. However, after training for multiple weeks, the result is still extremely low:

Is the result before PCAW supposed to be like this? Or maybe there is something wrong? I just follow the instruction:
python train.py
--config_path patchnetvlad/configs/train.ini
--cache_path=/path/to/your/desired/cache/folder
--save_path=/path/to/your/desired/checkpoint/save/folder
--dataset_root_dir=/path/to/your/mapillary/dataset
And there is no error reported.
Thank you so much for your time. Looking forward to hearing from you.

[Tobias-Fischer]

Hi, this could well be the case - please see #48.

Feel free to reopen if you have more questions.

[Jeff-Zilence]

Thank you for your reply. Is there any instruction on how to evaluate the PCAW model? It won't load correctly if I use the train.py

[Tobias-Fischer]

@StephenHausler - could you please help @Jeff-Zilence out?

[StephenHausler]

Hi @Jeff-Zilence,
There are two ways in which you can evaluate the PCAW model. The first is to use feature_extract.py and feature_match.py, you can use the provided dataset_imagenames files for mapillary cph and sf validation sets.

Alternatively you can still use train.py if you edit your local copies of train.py and val.py. First, edit lines 127 and 139 of train.py and set append_pca_layer=True. Then comment out line 227 of train.py (you can't train with the PCAW attached, but you can still do validation). Then in val.py, replace line 71 with:
vlad_encoding_pca = get_pca_encoding(model, vlad_encoding)
feat[indices.detach().numpy(), :] = vlad_encoding_pca.detach().cpu().numpy()

Happy to help if you have any more questions.

[Jeff-Zilence]

Hi @Jeff-Zilence, There are two ways in which you can evaluate the PCAW model. The first is to use feature_extract.py and feature_match.py, you can use the provided dataset_imagenames files for mapillary cph and sf validation sets.

Alternatively you can still use train.py if you edit your local copies of train.py and val.py. First, edit lines 127 and 139 of train.py and set append_pca_layer=True. Then comment out line 227 of train.py (you can't train with the PCAW attached, but you can still do validation). Then in val.py, replace line 71 with: vlad_encoding_pca = get_pca_encoding(model, vlad_encoding) feat[indices.detach().numpy(), :] = vlad_encoding_pca.detach().cpu().numpy()

Happy to help if you have any more questions.

Thank you so much for your help. I follow your instructions and fix multiple bugs. However, the performance is still far from the number reported in the paper. Is there anything wrong in the val.py? Or maybe other bug? I directly run validation without training PCAW. I have tried the pretrained PCAW4096 checkpoint and got the same results.

[StephenHausler]

I don't think there is anything wrong with these numbers you are getting, for straight NetVLAD. You won't reach the number reported in the paper for mapillary val (R@1 = 79.5%), because that was only achieved with local feature matching of the patch-netvlad descriptors.

Just thought I'd also mention that your earlier numbers without PCA are consistent with the numbers I also get with NetVLAD: #44

Also have a read of: #25
This issue explains how to replicate the Patch-NetVLAD results on Mapillary val.

[Jeff-Zilence]

Thank you for your reply. Do you mean the instruction is for training NetVLAD rather than PatchNetVLAD? How about the pretrained model? I am just wondering how to train a PatchNetVLAD and get the result reported in the paper. By the way, why does the training take multiple weeks?

[StephenHausler]

Yep, but the exact same model is used for both NetVLAD and Patch-NetVLAD. You just load the same weights into the Patch-NetVLAD layer instead and run the local feature matching, using feature_extract.py and feature_match.py.

It takes so long because there are ~1 million images per epoch. It's probably overkill though, the loss gets small well before 20 epochs with this many images per epoch. If you want faster training, you could edit the training code and only train on a random say 100k slice of images per epoch. The training can also be sped up with a larger GPU with more RAM, in which case you can increase the batch size.

[Jeff-Zilence]

Thank you for your reply. I have tried the feature_extract.py and feature_match.py, but it says that the ground-truth is missing. Is it possible to provide the ground-truth file for mapillary?