LoST? Appearance-Invariant Place Recognition for Opposite Viewpoints using Visual Semantics

This is the source code for the paper titled - "LoST? Appearance-Invariant Place Recognition for Opposite Viewpoints using Visual Semantics", pre-print available here.

An example output image showing Keypoint Correspondences:

Flowchart of the proposed approach:

If you find this work useful, please cite it as:
Sourav Garg, Niko Sunderhauf, and Michael Milford. LoST? Appearance-Invariant Place Recognition for Opposite Viewpoints using Visual Semantics. Proceedings of Robotics: Science and Systems XIV, 2018.
bibtex:

@article{garg2018lost,
title={LoST? Appearance-Invariant Place Recognition for Opposite Viewpoints using Visual Semantics},
author={Garg, Sourav and Suenderhauf, Niko and Milford, Michael},
journal={Proceedings of Robotics: Science and Systems XIV},
year={2018}
}

RefineNet's citation as mentioned on their Github page.

Setup and Run

Dependencies

• Ubuntu (Tested on 14.04)
• RefineNet
  • Required primarily for visual semantic information. Convolutional feature maps based dense descriptors are also extracted from the same.
  • A modified fork of RefineNet's code is used in this work to simultaneously store convolutional dense descriptors.
  • Requires Matlab (Tested on 2017a)
• Python (Tested on 2.7)
  • numpy (Tested on 1.11.1, 1.14.2)
  • scipy (Tested on 0.13.3, 0.17.1)
  • skimage (Minimum Required 0.13.1)
  • sklearn (Tested on 0.14.1, 0.19.1)
  • h5py (Tested on 2.7.1)
• Docker (optional, recommended, tested on 17.12.0-ce)
  • Official page for install instructions

Download

1. In your workspace, clone the repositories:

   git clone https://github.com/oravus/lostX.git
   cd lostX
   git clone https://github.com/oravus/refinenet.git
   

   NOTE: If you download this repository as a zip, the refineNet's fork will not get downloaded automatically, being a git submodule.
2. Download the Resnet-101 model pre-trained on Cityscapes dataset from here or here. More details on RefineNet's Github page.
   • Place the downloaded model's .mat file in the refinenet/model_trained/ directory.
3. If you are using docker, download the docker image:

   docker pull souravgarg/vpr-lost-kc:v1
   

Run

1. Generate and store semantic labels and dense convolutional descriptors from RefineNet's conv5 layer In the MATLAB workspace, from the refinenet/main/ directory, run:

   demo_predict_mscale_cityscapes
   

   The above will use the sample dataset from refinenet/datasets/ directory. You can set path to your data in demo_predict_mscale_cityscapes.m through variable datasetName and img_data_dir.
   You might have to run vl_compilenn before running the demo, please refer to the instructions for running refinenet in their official Readme.md

2. [For Docker users]
   If you have an environment with python and other dependencies installed, skip this step, otherwise run a docker container:

   docker run -it -v PATH_TO_YOUR_HOME_DIRECTORY/:/workspace/ souravgarg/vpr-lost-kc:v1 /bin/bash
   

   From within the docker container, navigate to lostX/lost_kc/ repository.
   -v option mounts the PATH_TO_YOUR_HOME_DIRECTORY to /workspace directory within the docker container.

3. Reformat and pre-process RefineNet's output from lostX/lost_kc/ directory:

   python reformat_data.py -p $PATH_TO_REFINENET_OUTPUT
   

   $PATH_TO_REFINENET_OUTPUT is set to be the parent directory of predict_result_full, for example, ../refinenet/cache_data/test_examples_cityscapes/1-s_result_20180427152622_predict_custom_data/predict_result_1/

4. Compute LoST descriptor:

   python LoST.py -p $PATH_TO_REFINENET_OUTPUT 
   

5. Repeat step 1, 3, and 4 to generate output for the other dataset by setting the variable datasetName to 2-s.

6. Perform place matching using LoST descriptors based difference matrix and Keypoint Correspondences:

   python match_lost_kc.py -n 10 -f 0 -p1 $PATH_TO_REFINENET_OUTPUT_1  -p2 $PATH_TO_REFINENET_OUTPUT_2
   

Note: Run python FILENAME -h for any of the python source files in Step 3, 4, and 6 for description of arguments passed to those files.

License

The code is released under MIT License.