EvalCrafter: Benchmarking and Evaluating Large Video Generation Models (CVPR 2024) 🎥📊

Project Page   ·   Leaderboard   ·   Gallery   ·   Paper@ArXiv   ·   EvalBoard

ECTV Dataset   ·   Human Feedback   ·   Prompt List

Welcome to EvalCrafter, a comprehensive evaluation toolkit for AI-generated videos. Our innovative framework assesses generative models across visual, content, and motion qualities using 17 objective metrics and subjective user opinions, providing a reliable ranking for state-of-the-art text-to-video generation models. Dive into the world of unified and user-centric evaluation with EvalCrafter! 🚀🌍📊

🔥 2023/10/22: Release prompt list at Prompt List! You can generate the resulting video and send it to vinthony@gmail.com for evaluation!

🔥 2024/01/10: Code and docker released!

🔥 2024/01/24: EvalCrafter Text-to-Video (ECTV) dataset released, around 10000 AI-generated videos in total, take a look at here!

🔥 2024/03/07: Dive into the exciting world of AI-Created Videos at our Gallery 🌌✨!

🔥 2024/04/07: Release the code of EvalBoard 🌌✨ for video comparison !

🔥 2024/04/11: Release the Human Feedback data, 8.6k feedback, cover 5 aspects like visual quality and t2v alignment! ！🚀🔥

🔥 2024/08/01: Environment setup using Conda is Supported！🛫🤘

🔆 Join our Discord to enjoy free text-to-video generation and more:

🔆 Watch our project for more details and findings.

Installation 💻

Clone the repository:

git clone https://github.com/evalcrafter/EvalCrafter
cd EvalCrafter

Data Preparation 📚

Generate videos of your model using the 700 prompts provided in prompt700.txt or ./prompts and organize them in the following structure:

/EvalCrafter/videos
├── 0000.mp4
├── 0001.mp4
├── 0002.mp4
├── 0003.mp4
├── 0004.mp4
...
└── 0699.mp4

Pretrained Models 🧠

Please download all checkpoints using

cd checkpoints
bash download.sh

Alternatively, you can follow ./checkpoints/README.md to download pretrained models for specific metrics.

Note: Please organize the pretrained models in this structure:

/EvalCrafter/checkpoints/
├── bert-base-uncased
├── blip2-opt-2.7b
├── ckpt
├── clip-vit-base-patch32
├── Dover
├── FlowNet2_checkpoint.pth.tar
├── pt_inception-2015-12-05-6726825d.pth
├── RAFT
├── stable-diffusion-xl-base-1.0
├── tsn_imagenet-pretrained-r50_8xb32-1x1x8-100e_kinetics400-rgb_20220906-2692d16c.pth
├── vgg_face_weights.h5
└── VideoMAE

Setup 🛠️

Download Docker Image 🐳

docker pull bruceliu1/evalcrafter:v1

Or, create a conda environment 🎮

conda env create -f environment.yml
conda activate EvalCrafter

cd ./metrics/RAFT/networks/resample2d_package
python setup.py install --user

Note that if you encounter this problem RuntimeError: GET was unable to find an engine to execute this computation when running the evaluation code, you may update Cuda to version Cuda 11.7 (Compatibility issue).

Usage 🚀

Running the Whole Pipeline

1. Run with command line:

   docker run --runtime=nvidia -it --shm-size "15G" -v $EC_path:$EC_path bruceliu1/evalcrafter:v1 \
      bash -c "source /opt/conda/bin/activate EvalCrafter \
         && bash $bash_file $EC_path $EC_path/videos"
   

   🔁 Please replace $EC_path, $bash_file, and $dir_videos with your local path to EvalCrafter, EvalCrafter/start.sh, and EvalCrafter/videos, respectively.

Alternatively, you can:

2. Enter the Docker container and run:

   docker run --runtime=nvidia -it --shm-size "15G" -v $EC_path:$EC_path bruceliu1/evalcrafter:v1 bash
   cd $EC_path
   bash start.sh $EC_path $dir_videos
   

3. If you use conda, you can directly run the evaluation code by:

   bash start.sh $EC_path $dir_videos
   

Running a Single Metric

1. 🔧 To test a specific metric, pick out the code for the metric in start.sh. For example, to test the Celebrity ID Score:

   docker run --runtime=nvidia -it --shm-size "15G" -v $EC_path:$EC_path bruceliu1/evalcrafter:v1 bash
   cd $EC_path
   cd /metrics/deepface
   python3 celebrity_id_score.py --dir_videos $dir_videos
   

2. If you use conda, you can directly run the evaluation code by:

   cd $EC_path
   cd /metrics/deepface
   python3 celebrity_id_score.py --dir_videos $dir_videos
   

EvalCrafter Gallery🌌

Please ref to EvalBoard to use it.

1. Clone the repository:

   git clone https://github.com/evalcrafter/EvalBoard
   

2. Navigate to the project directory:

   cd EvalBoard
   

3. Install the required dependencies:

   pip install -r requirements.txt
   

4. Start EvalBoard:

   python app.py
   

Acknowledgements 🙏

This work is based on the following open-source repositories:

• deepface
• DOVER
• mmaction2
• CLIP
• RAFT
• pytorch-gan-metrics
• SDXL
• PaddleOCR
• SAM-Track
• BILIP2
• HRS-Bench
• fast_blind_video_consistency

Citation

If you find this repository helpful, please consider citing it in your research:

@article{liu2023evalcrafter,
title={Evalcrafter: Benchmarking and evaluating large video generation models},
author={Liu, Yaofang and Cun, Xiaodong and Liu, Xuebo and Wang, Xintao and Zhang, Yong and Chen, Haoxin and Liu, Yang and Zeng, Tieyong and Chan, Raymond and Shan, Ying},
journal={arXiv preprint arXiv:2310.11440},
year={2023}
}

Know More About Video Generation at:

• VideoCrafter1: Open Diffusion Models for High-Quality Video Generation
• ScaleCrafter: Tuning-free Higher-Resolution Visual Generation with Diffusion Models
• TaleCrafter: Interactive Story Visualization with Multiple Characters