Learning Local Implicit Fourier Representation for Image Warping

This repository contains the official implementation for LTEW introduced in the following paper:

Learning Local Implicit Fourier Representation for Image Warping (ECCV 2022)

Installation

Our code is based on Ubuntu 20.04, pytorch 1.10.0, CUDA 11.3 (NVIDIA RTX 3090 24GB, sm86) and python 3.6.

We recommend using conda for installation:

conda env create --file environment.yaml
conda activate ltew

Then, please install pysrwarp as described in SRWarp.

git clone https://github.com/sanghyun-son/pysrwarp
cd pysrwarp
make

If your CUDA compatibility is sm86, modify cuda/Makefile before make.

vi cuda/Makefile

Quick Start

0. Download datasets.

Table 3: Homography transformation

• DIV2K: DIV2KW (isc), DIV2KW (osc)

• Benchmark datasets: Set5W (isc), Set5W (osc), Set14W (isc), Set14W (osc), B100W (isc), B100W (osc), Urban100W (isc), Urban100W (osc)

Table 5: Symmetric-scale SR

• B100 dataset with transformation: B100

mkdir load and put the datasets as follows:

load
├── benchmark
│   ├── B100
│   │   ├── HR
│   │   ├── LR_bicubic
│   │   ├── LR_bicubic_Transform
│   │   ├── LR_warp_in_scale
│   │   └── LR_warp_out_of_scale
│   ├── Set14
│   │   ├── HR
│   │   ├── LR_bicubic
│   │   ├── LR_warp_in_scale
│   │   └── LR_warp_out_of_scale
│   ├── Set5
│   │   ├── HR
│   │   ├── LR_bicubic
│   │   ├── LR_warp_in_scale
│   │   └── LR_warp_out_of_scale
│   └── Urban100
│       ├── HR
│       ├── LR_bicubic
│       ├── LR_warp_in_scale
│       └── LR_warp_out_of_scale
├── div2k
│   ├── DIV2K_train_HR
│   ├── DIV2K_train_LR_bicubic
│   ├── DIV2K_valid_HR
│   ├── DIV2K_valid_LR_bicubic
│   ├── DIV2K_valid_LR_warp_in_scale
│   └── DIV2K_valid_LR_warp_ouf_of_scale

1. Download pre-trained models.

Model	Download
LTEW-EDSR-baseline	Google Drive
LTEW-RRDB	Google Drive
LTEW-RCAN	Google Drive

2. Reproduce experiments.

Table 1, 2: Asymmetric-scale SR

bash ./scripts/test-benchmark-asym.sh save/rcan-lte.pth 0

Table 3: Homography transformation

bash ./scripts/test-div2k-warp.sh ./save/rrdb-lte-warp.pth 0

bash ./scripts/test-benchmark-warp.sh ./save/rrdb-lte-warp.pth 0

Table 5: Symmetric-scale SR

bash ./scripts/test-b100-sym-w-lte.sh save/rcan-lte.pth 0

bash ./scripts/test-b100-sym-w-ltew.sh save/rrdb-lte-warp.pth 0

Train & Test

Asymmetric-scale SR

Train: CUDA_VISIBLE_DEVICES=0 python train_lte.py --config configs/train-div2k/train_rcan-lte.yaml --gpu 0

Test: bash ./scripts/test-benchmark-asym.sh save/_train_rcan-lte/epoch-last.pth 0

Homography transformation

Train: CUDA_VISIBLE_DEVICES=0,1 python train_ltew.py --config configs/train-div2k/train_rrdb-lte-warp.yaml --gpu 0,1

Test: bash ./scripts/test-benchmark-warp.sh ./save/_train_rrdb-lte-warp/epoch-last.pth 0

Model	Training time (# GPU)
EDSR-baseline-LTEW	39h (1 GPU)
RRDB-LTEW	106h (2 GPU)
RCAN-LTEW	130h (1 GPU)

We use NVIDIA RTX 3090 24GB for training.

Fourier Space

The script Eval-Fourier-Feature-Space is used to generate the paper plots.

Demo ERP

Download the StreetLearn.

Then, we downsample HR ERP images by a factor of 4 and then project to a size of 832 X 832 with a field of view (FOV) 120-deg for Fig.9.

python demo.py --input ./load/streetlearn/LR_bicubic/kc1Ppxk2yKIsNV9UCvOlbg.png --mode erp2pers --model save/edsr-baseline-lte-warp.pth --FOV 120 --THETA 0 --PHI 0 --resolution 832,832 --output ./save_image/erp2pers-kc1Ppxk2yKIsNV9UCvOlbg.png --gpu 0

For perspective view -> ERP,

python demo.py --input ./save_image/erp2pers-kc1Ppxk2yKIsNV9UCvOlbg.png --mode pers2erp --model save/edsr-baseline-lte-warp.pth --FOV 120 --THETA 0 --PHI 0 --resolution 832,1664 --output ./save_image/pers2erp-kc1Ppxk2yKIsNV9UCvOlbg.png --gpu 0

For ERP -> fisheye view,

python demo.py --input ./load/streetlearn/LR_bicubic/kc1Ppxk2yKIsNV9UCvOlbg.png --mode erp2fish --model save/edsr-baseline-lte-warp.pth --FOV 180 --THETA 0 --PHI 0 --resolution 832,832 --output ./save_image/erp2fish-kc1Ppxk2yKIsNV9UCvOlbg.png --gpu 0

For fisheye view -> ERP,

python demo.py --input ./save_image/erp2fish-kc1Ppxk2yKIsNV9UCvOlbg.png --mode fish2erp --model save/edsr-baseline-lte-warp.pth --FOV 180 --THETA 0 --PHI 0 --resolution 832,1664 --output ./save_image/fish2erp-kc1Ppxk2yKIsNV9UCvOlbg.png --gpu 0

For fisheye view -> perspective view,

python demo.py --input ./save_image/erp2fish-kc1Ppxk2yKIsNV9UCvOlbg.png --mode fish2pers --model save/edsr-baseline-lte-warp.pth --FOV 120 --THETA 0 --PHI 0 --resolution 832,832 --output ./save_image/fish2pers-kc1Ppxk2yKIsNV9UCvOlbg.png --gpu 0

Demo MARS

The directory ./fig contains full-resolution MARS images.

Citation

If you find our work useful in your research, please consider citing our paper:

@article{ltew-jaewon-lee,
  title={Learning Local Implicit Fourier Representation for Image Warping},
  author={Jaewon Lee, Kwang Pyo Choi, Kyong Hwan Jin},
  journal={ECCV},
  year={2022}
}

Acknowledgements

This code is built on LIIF and SRWarp. We thank the authors for sharing their codes.