[IROS2022] STUN: Self-Teaching Uncertainty Estimation for Place Recognition

0. Environment Setup ⚙️

• Ubuntu 18.04, python 3.8, A100
• PyTorch 1.8.1 + CUDA 11.1

1. Download the Pittsburgh dataset and pretrained models 📨

bash ./download.sh

the files will be downloaded and saved in the following folders:

pittsburgh
├── database
├── query
└── structure

logs
├── student_contrast
├── student_quadruplet
├── student_triplet
└── teacher_triplet

2. Evaluate the pretrained models 🔍

# STUN
python main.py  --resume=logs/student_triplet/ckpt.pth.tar

# STUN (Constrast)
python main.py --resume=logs/student_constrast/ckpt.pth.tar

# STUN (Quadruplet)
python main.py --resume=logs/student_quadruplet/ckpt.pth.tar

# Standard Triplet
python main.py --phase=test_tea	 --resume=logs/teacher_triplet/ckpt.pth.tar

3. Plot results 📈

python vis_results.py
# you can plot results of different models by populate the NETWORK variable.

4. Train and evaluate STUN from scratch 🧭

# train the teacher net
python main.py --phase=train_tea --loss=tri

# train the student net supervised by the pretrained teacher net
python main.py --phase=train_stu --resume=[teacher_net_xxx/ckpt_best.pth.tar]

After analyzing empirical figures, we found the correlation between recall@N and uncertainty level evolve into a sensible trend after 30 epochs. But ECE (Expected Calibration Error) will diverge if the student network is excessively trained. As a result, we focused our examination on the model's performance from epoch=30 to epoch=35 and chose the one with the lowest ECE.

# evaluate 
./eval_batch.sh

If you find our work useful, please consider citing:

@INPROCEEDINGS{stun_cai,
  author={Cai, Kaiwen and Lu, Chris Xiaoxuan and Huang, Xiaowei},
  booktitle={2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)}, 
  title={STUN: Self-Teaching Uncertainty Estimation for Place Recognition}, 
  year={2022},
  volume={},
  number={},
  pages={6614-6621},
  doi={10.1109/IROS47612.2022.9981546}}