Multimodal Object Detection via Probabilistic Ensembling

ECCV 2022 Oral presentation

[project page] [code] [video demo] [paper] [models] [results]

The results of ProbEn are released! (KAIST / FLIR)

Authors: Yi-Ting Chen^*, Jinghao Shi^*, Zelin Ye^*, Christoph Mertz, Deva Ramanan^#, Shu Kong^#

For installation, please check INSTALL.md.

Usage

We provide the training, testing, and visualization code of thermal-only, early-fusion, middle-fusion and Bayesian fusion. Please change the setting for different fusion methods in the code.

Training:

python demo/FLIR/demo_train_FLIR.py

Test mAP:

python demo/FLIR/demo_mAP_FLIR.py

Visualize predicted boxes:

python demo/FLIR/demo_draw_FLIR.py    

Probabilistic Ensembling:

First, you should save predictions from different models using demo_FLIR_save_predictions.py

# Example thermal only
python demo/FLIR/demo_FLIR_save_predictions.py --dataset_path /home/jamie/Desktop/Datasets/FLIR/val --fusion_method thermal_only --model_path trained_models/FLIR/models/thermal_only/out_model_thermal_only.pth

# Example early fusion
python demo/FLIR/demo_FLIR_save_predictions.py --dataset_path /home/jamie/Desktop/Datasets/FLIR/val --fusion_method early_fusion --model_path trained_models/FLIR/models/early_fusion/out_model_early_fusion.pth

# Example middle fusion
python demo/FLIR/demo_FLIR_save_predictions.py --dataset_path /home/jamie/Desktop/Datasets/FLIR/val --fusion_method middle_fusion --model_path trained_models/FLIR/models/middle_fusion/out_model_middle_fusion.pth

Then, you can change and load the predictions in demo_probEn.py

python demo/FLIR/demo_probEn.py --dataset_path /home/jamie/Desktop/Datasets/FLIR/val --prediction_path out/  --score_fusion max --box_fusion argmax

For more example usage, please check run.sh file.

If you find our model/method/dataset useful, please cite our work (arxiv manuscript):

@inproceedings{RGBT-detection,
  title={Multimodal Object Detection via Probabilistic Ensembling},
  author={Chen, Yi-Ting and Shi, Jinghao and Mertz, Christoph and Kong, Shu and Ramanan, Deva},
  booktitle={European Conference on Computer Vision (ECCV)},
  year={2022}
}