PSFusion

This is official Pytorch implementation of "Rethinking the necessity of image fusion in high-level vision tasks: A practical infrared and visible image fusion network based on progressive semantic injection and scene fidelity"

@article{TANG2022SuperFusion,
  title={SuperFusion: A versatile image registration and fusion network with semantic awareness},
  author={Tang, Linfeng and Deng, Yuxin and Ma, Yong and Huang, Jun and Ma, Jiayi},
  journal={IEEE/CAA Journal of Automatica Sinica},
  volume={9},
  number={12},
  pages={2121--2137},
  year={2022},
  publisher={IEEE}
}

Framework

The overall framework of the proposed PSFusion.

Recommended Environment

• torch 1.10.0
• cudatoolkit 11.3.1
• torchvision 0.11.0
• kornia 0.6.5
• pillow 8.3.2

To Test

1. Downloading the pre-trained checkpoint from best_model.pth and putting it in ./results/PSFusion/checkpoints.
2. Downloading the MSRS dataset from MSRS and putting it in ./datasets.
3. python test.py --dataroot=./datasets/MSRS --dataset_name=MSRS --resume=./results/PSFusion/checkpoints/best_model.pth

If you need to test other datasets, please put the dataset according to the dataloader and specify --dataroot and --dataset-name

To Train

Before training PSFusion, you need to download the pre-processed MSRS dataset MSRS and putting it in ./datasets.

Then running python train.py --dataroot=./datasets/MSRS --name=PSFusion

Motivation

Comparison of fusion and segmentation results.

Comparison of the computational complexity for semantic segmentation.

Network Architecture

The architecture of the superficial detail fusion module (SDFM) based on the channel-spatial attention mechanism.

The architecture of the profound semantic fusion module (PSFM) based on the cross-attention mechanism.

To Segmentation

BANet

Segmentation

SegNeXt

Experiments

Qualitative fusion results

Qualitative comparison of PSFusion with 9 state-of-the-art methods on the **MSRS** dataset.

Qualitative comparison of PSFusion with 9 state-of-the-art methods on the **M3FD** dataset.

Quantitative fusion results

Quantitative comparisons of the six metrics on 361 image pairs from the MSRS dataset. A point (x, y) on the curve denotes that there are (100*x)% percent of image pairs which have metric values no more than y.

Quantitative comparisons of the six metrics on 300 image pairs from the M3FD dataset.

Segmentation comparison

Segmentation results of various fusion algorithms on the MSRS dataset.

Per-class segmentation results on the MSRS dataset.

Potential of image-level fusion for high-level vision tasks

Segmentation results of feature-level fusion-based multi-modal segmentation algorithms and our image-level fusion-based solution on the MFNet dataset.

Per-class segmentation results of image-level fusion and feature-level fusion on the MFNet dataset.

If this work is helpful to you, please cite it as：

@article{TANG2022SuperFusion,
  title={SuperFusion: A versatile image registration and fusion network with semantic awareness},
  author={Tang, Linfeng and Deng, Yuxin and Ma, Yong and Huang, Jun and Ma, Jiayi},
  journal={IEEE/CAA Journal of Automatica Sinica},
  volume={9},
  number={12},
  pages={2121--2137},
  year={2022},
  publisher={IEEE}
}