Skip to content

Latest commit

 

History

History
120 lines (91 loc) · 4.64 KB

README.md

File metadata and controls

120 lines (91 loc) · 4.64 KB
Raw

Channel Attention Is All You Need for Video Frame Interpolation

Myungsub Choi, Heewon Kim, Bohyung Han, Ning Xu, Kyoung Mu Lee

2nd place in [AIM 2019 ICCV Workshop] - Video Temporal Super-Resolution Challenge

Project | Paper-AAAI (Download the paper [here] in case the AAAI link is broken) | Poster

Paper

Directory Structure

project
│   README.md
|   run.sh - main script to train CAIN model
|   run_noca.sh - script to train CAIN_NoCA model
|   test_custom.sh - script to run interpolation on custom dataset
|   eval.sh - script to evaluate on SNU-FILM benchmark
|   main.py - main file to run train/val
|   config.py - check & change training/testing configurations here
|   loss.py - defines different loss functions
|   utils.py - misc.
└───model
│   │   common.py
│   │   cain.py - main model
|   |   cain_noca.py - model without channel attention
|   |   cain_encdec.py - model with additional encoder-decoder
└───data - implements dataloaders for each dataset
│   |   vimeo90k.py - main training / testing dataset
|   |   video.py - custom data for testing
│   └───symbolic links to each dataset
|       | ...

Dependencies

Current version is tested on:

  • Ubuntu 18.04
  • Python==3.7.5
  • numpy==1.17
  • PyTorch==1.3.1, torchvision==0.4.2, cudatoolkit==10.1
  • tensorboard==2.0.0 (If you want training logs)
  • opencv==3.4.2
  • tqdm==4.39.0
# Easy installation (using Anaconda environment)
conda create -n cain
conda activate cain
conda install python=3.7
conda install pip numpy
conda install pytorch torchvision cudatoolkit=10.1 -c pytorch
conda install tqdm opencv tensorboard

Model

Dataset Preparation

  • We use Vimeo90K Triplet dataset for training + testing.
    • After downloading the full dataset, make symbolic links in data/ folder :
      • ln -s /path/to/vimeo_triplet_data/ ./data/vimeo_triplet
    • Then you're done!
  • For more thorough evaluation, we built SNU-FILM (SNU Frame Interpolation with Large Motion) benchmark.
    • Download links can be found in the project page.
    • Also make symbolic links after download :
      • ln -s /path/to/SNU-FILM_data/ ./data/SNU-FILM
    • Done!

Usage

Training / Testing with Vimeo90K dataset

  • First make symbolic links in data/ folder : ln -s /path/to/vimeo_triplet_data/ ./data/vimeo_triplet
  • For training: CUDA_VISIBLE_DEVICES=0 python main.py --exp_name EXPNAME --batch_size 16 --test_batch_size 16 --dataset vimeo90k --model cain --loss 1*L1 --max_epoch 200 --lr 0.0002
  • Or, just run ./run.sh
  • For testing performance on Vimeo90K dataset, just add --mode test option
  • For testing on SNU-FILM dataset, run ./eval.sh
    • Testing mode (choose from ['easy', 'medium', 'hard', 'extreme']) can be modified by changing --test_mode option in eval.sh.

Interpolating with custom video

  • Download pretrained models from [Here]
  • Prepare frame sequences in data/frame_seq
  • run test_custom.sh

Results

Video

Video

Citation

If you find this code useful for your research, please consider citing the following paper:

@inproceedings{choi2020cain,
    author = {Choi, Myungsub and Kim, Heewon and Han, Bohyung and Xu, Ning and Lee, Kyoung Mu},
    title = {Channel Attention Is All You Need for Video Frame Interpolation},
    booktitle = {AAAI},
    year = {2020}
}

Acknowledgement

Many parts of this code is adapted from:

We thank the authors for sharing codes for their great works.