UMFusion

Unsupervised Misaligned Infrared and Visible Image Fusion via Cross-Modality Image Generation and Registration [IJCAI2022 Oral Presentation]

By Di Wang, Jinyuan Liu, Xin Fan, and Risheng Liu

Updates

[2022-07-14] The pretrained models of registration network (MRRN) and fusion network (DIFN) are available!
[2022-06-21] The CPSTN is available!
[2022-05-30] The Chinese translation of our paper is available, please enjoy it! [中译版本]
[2022-05-25] Our paper is available online! [arXiv version]

Requirements

• CUDA 10.1
• Python 3.6 (or later)
• Pytorch 1.6.0
• Torchvision 0.7.0
• OpenCV 3.4
• Kornia 0.5.11

Data preparation

1. You can obtain deformation infrared images for training/testing process by

      cd ./data
      python get_test_data.py

In 'Trainer/train_reg.py', deformable infrared images are generated in real time by default during training.

2. You can obtain self-visual saliency maps for training IVIF fusion by

      cd ./data
      python get_svs_map.py

Get start

1. You can use the pseudo infrared images [link code: qqyj] generated by our CPSTN to train/test the registration process:

      cd ./Trainer
      python train_reg.py
   
      cd ./Test
      python test_reg.py

Please download the pretrained model (code: hk25) of the registration network MRRN.

2. If you want to generate pseudo-infrared images using our CPSTN for other datasets, you can directly run following commands:

   ## testing
      cd ./CPSTN
      python test.py --dataroot datasets/rgb2ir/RoadScene/testA --name rgb2ir_paired_Road_edge_pretrained --model test --no_dropout --preprocess none
   
   ## training
      cd ./CPSTN
      python train.py --dataroot ./datasets/rgb2ir/RoadScene --name rgb2ir_paired_Road_edge --model cycle_gan --dataset_mode unaligned

The training and testing data of our CPSTN can be downloaded from: datasets (code: u386)

Please download the pretrained model (code: i9ju) of CPSTN and put it into folder './CPSTN/checkpoints/pretrained/'

3. If you tend to train Registration and Fusion processes separately, You can run following commands:

      cd ./Trainer
      python train_reg.py
   
      cd ./Trainer
      python train_fuse.py

The corresponding test code 'test_reg.py' and 'test_fuse.py' can be found in 'Test' folder. Please download the pretrained model (code: 0rbm) of fusion network DIFN.

4. If you tend to train Registration and Fusion processes jointly, You can run following command:

       cd ./Trainer
       python train_reg_fusion.py

The corresponding test code 'test_reg_fusion.py' can be found in 'Test' folder.

Dataset

Please download the following datasets:

• RoadScene
• TNO

Experimental Results

Please download the pseudo infrared images generated by our CPSTN:

• Fake_infrared_images (code: qqyj)

Please download the registered infrared images by our UMF:

• Registered_results on RoadScene (code: 4cx2)
• Registered_results on TNO (code: 2edi)

Please download the fused images by our UMF:

• Fused_results on RoadScene (code: 1zuu)
• Fused_results on TNO (code: 22gc)

Related Projects

• IMF (An improved version of the UMF, accepted to IEEE TCSVT 2024)

Citation

@inproceedings{UMF,
	author    = {Di Wang and
	Jinyuan Liu and
	Xin Fan and
	Risheng Liu},
	title     = {Unsupervised Misaligned Infrared and Visible Image Fusion via Cross-Modality Image Generation and Registration},
	booktitle = {IJCAI},
	pages     = {3508--3515},
	year      = {2022}
}