Binbin Xu · Andrew J. Davison · Stefan Leutenegger
This is the official repository of our RA-L 2021 paper:
Deep Probabilistic Feature-metric Tracking,
Binbin Xu, Andrew J. Davison, Stefan Leutenegger,
IEEE Robotics and Automation Letters (RA-L), Vol. 6, No. 1, pp. 223-230, 2021 (ICRA 2021 presentation)
Best Paper Honorable Mention Award
[Paper] [Video]
You can reproduce the setup by using our anaconda environment configurations. We have provided an Makefile to help you install the environment.
make install
Everytime before you run, activate the environment inside the repo folder
source .anaconda3/bin/activate
The pre-trained network weights can be downloaded at here.
TUM RGBD Dataset: Download the dataset from TUM RGBD to '$YOUR_TUM_RGBD_DIR'. Create a symbolic link to the data directory as
ln -s $YOUR_TUM_RGBD_DIR code/data/data_tum
MovingObjects3D Dataset Download the dataset from MovingObjs3D to '$YOUR_MOV_OBJS_3D_DIR'. Create a symbolic link to the data directory as
ln -s $YOUR_MOV_OBJS_3D_DIR code/data/data_objs3D
Custom Dataset You can also use your own dataset. Our work use the above two datasets for training and deployed the trained weights on scannet and our self-collected dataset. Please refer to the ScanNet and VaryLighting.py for the custom dataloading.
To run the full training and evaluation, please follow the steps below.
Train example with TUM RGBD dataset:
./scripts/train_tum_rgbd.sh
To check the full training setting, run the help config as
python train.py --help
Train example with MovingObjects3D: Camera egocentric motion is dfifferent from the object-centric motion estimation and thus we provide a separate training script for the MovingObjects3D dataset. All the same as the last one only except changing the dataset name. You can also use our provided script to train the model.
./scripts/train_moving_objs3d.sh
Run the pretrained model: If you have set up the dataset properly with the datasets, you can run the learned model with the checkpoint we provided in the trained model directory.
./scripts/eval_tum_rgbd.sh
You can substitute the trajectory, the keyframe and the checkpoint file. The training and evaluation share the same config setting. To check the full setting, run the help config as
python evaluate.py --help
Results: The evaluation results will be generated automatically in both '.pkl' and '*.csv' in the folder 'test_results/'.
Run comparisons: We also provide the scripts to run the comparisons with the classic RGBD and ICP methods from Open3D. Please refer to rgbd_odometry.py and ICP.py for the details accordingly.
We can combine our proposed feature-metric tracking with the geometric tracking methods to achieve better performance. We provide the scripts to run the joint tracking with the ICP methods.
./scripts/train_tum_feature_icp.sh
It is achieved by using the trained feature-metric network weights as the initialization and combing with the ICP methods as the refinement.
The evaluation script is also provided as
./scripts/eval_tum_feature_icp.sh
Please note this is a prototype version of our visual odometry frontend. It mainly serves as a demo to show the performance of our method.
./scripts/run_kf_vo.sh
To visualise the keyframe tracking in the paper, add the argument --vo_type keyframe --two_view to the above script.
To check the full setting, run the help config as
python code/experiments/kf_vo.py --help
Convergence basin analysis for the keyframe visual odometry is also provided. Check the script scripts/run_kf_vo_cb.sh for more details.
@article{Xu:etal:RAL2021,
author = {Binbin Xu and Andrew Davison and Stefan Leutenegger},
journal = {{IEEE} Robotics and Automation Letters ({RAL})},
title = {Deep Probabilistic Feature-metric Tracking},
year={2021},
volume = {6},
number = {1},
pages = {223 - 230},
}Please cite the paper if you found our provided code useful for you.
This repo is BSD 3-Clause Licensed. Part of its code is from Taking a Deeper Look at the Inverse Compositional Algorithm, which is MIT licensed. We thank the authors for their great work.
Copyright © 2020-2021 Smart Robotics Lab, Imperial College London
Copyright © 2020-2021 Binbin Xu
Binbin Xu (b.xu17@imperial.ac.uk)