clic2021-devkit

Challenge homepage: compression.cc.

Downloading the Video challenge Data

To download all files, run:

bash download.sh path/to/data

It will create a folder path/to/data and extract all frames there, into a structure like:

video1/
    video1_frame1_y.png
    video1_frame1_u.png
    video1_frame1_v.png
    video1_frame2_y.png
    video1_frame2_u.png
    video1_frame2_v.png
    ...
video2/
    video2_frame1_y.png
    video2_frame1_u.png
    video2_frame1_v.png
    ...

For this, one of gsutil, wget, or curl must be available. gsutil is probably the most efficient way.

To download only some videos, use --max_vides: bash download.sh path/to/data --max_videos 10

NOTE: The script first downloads all vidoes as .zip files, resulting in 250GB+ of data. Then all zips are decompressed one by one and subsequently deleted. If you interrupt the script while unpacking, and later re-run it, it will re-download those that were already unpacked. To prevent this at the expense of more hard-drive space used, you can keep the zip files by passing --no_delete_zip.

Perceptual Challenge

This development kit is only provided as an example of what is expected from participants. It is in no way intended to contain data representative of the final challenge simply because that is not possible. The final test set will be created from the files uploaded by the participants in the compression challenge, and as a result it’s simply impossible for us to provide data which will match that distribution in the validation set.

You will first need to download the data from here. We recommend using wget to download this file as it supports resuming the download gets interrupted.

The first step will be to unzip the data:

unzip clic_2021_perceptual_valid.zip

Once unzipped, this should contain 2730 PNG files. These are of the same size (768x768) that will be shown to human raters. The test set will be similarly distributed as PNG files. We don’t yet know how many there will be.

In addition to the PNG files, there are two important CSV files:

validation.csv - this file is an example CSV file which contains the triplet PNG files that are used in the evaluation. The goal of your binary is to take this file as input, and generate a CSV file containing the same triplets, and an additional column which should contain either a 0, or a 1. The last column should have a 0 if the pair <O,A> will be preferred by humans to the pair <O,B>. Otherwise the last column should contain a 1.

oracle.csv - this file contains the “ground truth” and we provide it as a way to evaluate your metric by running “eval_csv.py”. This file will NOT be given to participants as part of the test set.

Here is a short description of the scripts related to the pereceptual challenge:

eval_csv.py: this script takes two arguments which are the oracle CSV (see below), and the CSV file generated by your metric.

Example usage:

python3 eval_csv.py --oracle_csv oracle.csv --eval_csv psnr.csv

For convenience (and perhaps as a very naive starting point), we provide psnr.py which is a very simple script which can be used to produce outputs that are compatible with the evaluation script. This takes the validation.csv file (specified as an argument) and produces output compatible with eval_csv.py.