ReF-LDM: A Latent Diffusion Model for Reference-based Face Image Restoration

NeurIPS 2024

🚩 Updates

• Release FFHQ-Ref Dataset
• Release ReF-LDM model and inference script

📌 TODO

• inference script for testing datasets

ReF-LDM Model

ReF-LDM leverages a flexible number of reference images to restore a low-quality (LQ) face image into a high-quality (HQ) one.

Visual Results

Architecture

Model Inference

• 👉 Download model

• Place model weights refldm.ckpt and vqgan.ckpt under ckpts/

• Run inference.py

  python inference.py --ddim_step 50 --output_path result.png --lq_path assets/demo/lq.png --ref_paths assets/demo/ref0.png assets/demo/ref1.png assets/demo/ref2.png assets/demo/ref3.png

FFHQ-Ref Dataset

👉 Download data

FFHQ-Ref/
│
├── reference_mapping/
│   ├── train_references.csv
│   ├── val_references.csv
│   └── test_references.csv
│
├── id_based_ffhq_split/
│   ├── train_image.txt
│   ├── val_image.txt
│   └── test_image.txt
│
└── test_images/
    ├── severe_degrad/
    └── moderate_degrad/

Dataset Contents

FFHQ-Ref Dataset

FFHQ-Ref contains 20,405 high-quality face images with corresponding reference images. It is constructed from the 70,000 images of the FFHQ dataset using facial identity labels predicted by ArcFace.

• High-quality images
  • Download from FFHQ dataset (images1024x1024/)
• reference_mapping/
  • CSV files that list target images and their corresponding reference images
• id_based_ffhq_split/
  • Text files that list images for identity-based train/val/test splits of FFHQ dataset (70,000 images)
  • Why is this needed? Previous works randomly split the FFHQ dataset, which resulted in images of the same person being distributed across both training and evaluation sets. We provide identity-based data splits to address this issue.
• test_images/
  • Low-quality test images with two degradation levels

CelebA-Test-Ref Dataset

An additional testing dataset for reference-based face restoration, containing 2,533 images with corresponding reference images.

• High-quality images
  • Download from CelebAMask-HQ (CelebAMask-HQ/CelebA-HQ-img/)
• test_references.csv
  • Lists target images and their corresponding reference images
• celeba_test_images/
  • Contains low-quality test images and high-quality ground truth images

Acknowledgments

• The code of ReF-LDM model is modified from CompVis/latent-diffusion.
• All images in the FFHQ-Ref dataset come from NVlabs/ffhq-dataset.
• We used the ArcFace model from deepinsight/insightface to find reference images.
• We used dchen236/FairFace to analyze the age, gender, and race distribution of the FFHQ-Ref dataset.

License and Usage

The FFHQ-Ref dataset and ReF-LDM model are provided for non-commercial research purposes only. Any commercial use is strictly prohibited.

Citation

@inproceedings{hsiao2024refldm,
  title={ReF-LDM: A Latent Diffusion Model for Reference-based Face Image Restoration},
  author={Chi-Wei Hsiao and Yu-Lun Liu and Cheng-Kun Yang and Sheng-Po Kuo and Yucheun Kevin Jou and Chia-Ping Chen},
  journal={Advances in Neural Information Processing Systems},
  year={2024}
}