DiffIR2VR-Zero: Zero-Shot Video Restoration with Diffusion-based Image Restoration Models

DiffIR2VR-Zero is a zero-shot video restoration method that uses pretrained image diffusion models to handle video restoration tasks such as denoising and up to 8x super-resolution.

---

Overview

Installation

# clone this repo
git clone https://github.com/jimmycv07/DiffIR2VR-Zero.git
cd DiffIR2VR-Zero

# create environment
conda create -n diffir2vr python=3.10
conda activate diffir2vr
pip install -r requirements.txt

Pretrained Weights

Please place the pretrained weights like the following folder structure. You will need to download the GMFlow weights manually; the rest will be downloaded automatically during inference.

weights
└─── gmflow_sintel-0c07dcb3.pth
└─── v2.pth
└─── v2-1_512-ema-pruned.ckpt

Inference

We provide some examples of inference commands; for additional arguments, please refer to inference.py.

Blind Video Denoising

python -u inference.py \
--version v2 \
--task dn \
--upscale 1 \
--cfg_scale 4.0 \
--batch_size 10 \
--input inputs/noise_50/flamingo \
--output results/Denoise/flamingo  \
--config configs/inference/my_cldm.yaml \
--final_size "(480, 854)" \
--merge_ratio "(0.6, 0)" \
--better_start 

Blind Video Super-Resolution

python -u inference.py \
--version v2 \
--task sr \
--upscale 4 \
--cfg_scale 4.0 \
--batch_size 10 \
--input inputs/BDx4/rhino \
--output results/SR/rhino  \
--config configs/inference/my_cldm.yaml \
--final_size "(480, 854)" \
--merge_ratio "(0.6, 0)" 

Citation

Please cite us if our work is useful for your research.

@article{yeh2024diffir2vr,
    title={DiffIR2VR-Zero: Zero-Shot Video Restoration with Diffusion-based Image Restoration Models},
    author={Chang-Han Yeh and Chin-Yang Lin and Zhixiang Wang and Chi-Wei Hsiao and Ting-Hsuan Chen and Yu-Lun Liu},
    journal={arXiv},
    year={2024}
}

Acknowledgement

This project is based on DiffBIR and VidToMe. Thanks for their awesome work.