deepfakes_classification

This repository provides the official Python implementation of Deepfakes Detection with Metric Learning accepted at 8th International Workshop on Biometrics and Forensics. Medium blog post is shared here: deepfakes-classification-via-metric-learning

Table of Contents

• Requirements
• Celeb-DF
• Face-forensics
• Results
• Citation
• Notes

Requirements

Tested on Python 3.6.x and Keras 2.3.0 with TF backend version 1.14.0.

• Numpy (1.16.4)
• OpenCV (4.1.0)
• Pandas (0.25.3)
• Scikit-learn (0.22.1)
• facenet-pytorch (2.0.1)
• PyTorch (1.2.0)

Getting Started

• Install the required dependencies:

pip install -r requirements.txt

• frames_extraction.py - Extract frames from videos
• face_extraction.py - Extract faces from frames/videos for training purpose
• train_data_prepare.py - Selecting the first n frames per video and then saving it to numpy file for CNN training

python train_data_prepare.py -img_size 160 -fpv 25

-  [-img_size] IMG_SIZE, Resize face image size
-  [-fpv] FRAMES_PER_VIDEO, Number of frames per video to consider

• train_CNN.py - Train on 2D CNN - XceptionNet model

python train_CNN.py -e 20 -m xception -b 32

-  [-e] EPOCHS, Number of epochs
-  [-m] MODEL_NAME, Imagenet model to train
-  [-b] BATCH_SIZE, Batch size

• evaluate_CNN.py - Evaluate the testing video accuracy using CNN
• train_C3D.py - Train on Convolutional 3D architecture

python train_C3D.py -e 15 -m c3d -b 32

-  [-e] EPOCHS, Number of epochs
-  [-m] MODEL_NAME, conv3d/ c3d model
-  [-b] BATCH_SIZE, Batch size

• evaluate_C3D.py - Evaluate videos using 3D CNN architecture

• feature_extractor.py - Extract features for recurrence networks

• train_CNN_LSTM.py - Train the LSTM/GRU (BiDirectional)/ Temporal models

• evaluate_CNN_LSTM.py - Evaluate the recurrence models

• train_triplets_semi_hard.py - Train the triplets of face embedding vectors and then train ML classifiers such as SGD, Random Forest, etc. to classify feature vectors

• evaluate_triplets.py - Evaluate the testing video embeddings uing trained ML classifiers

Celeb-DF

It contains high resolution videos, with 5299/712 training distribution and 340/178 videos in testing distribution as real/fake videos. With frame rate 5, there are approximately 70K frames generated.

Although Celeb-DF face quality is better than FaceForensics++ c-40 videos, training directly on whole frames is not useful. Therefore, we extracted faces from frames and then utilised that for classification. Data imbalance plays a huge role that affects the weights of network. In our case, it was 7:1. We applied bagging and boosting algorithm. So, the dataset was divided into 7 chunks of 1400 videos approximately: 700 fake and 700 real. It was trained on each distribution and then performance was boosted by max voting all the predictions.

Frames contains a lot of noise and we have to focus on face. We used facenet model to extract faces from the whole video (can be done directly using videos or after extraction of frames), and then we trained XceptionNet for 50 epochs with EarlyStopping (patience=10) and ModelCheckpoint to save only the best mdoel by tracking the val_loss. We achieve the accuracy of 96% and after boosting accuracy improves to 98%.

TSNE plot before and after training using frames only (2D-CNN- Xception)

TSNE plot before and after training using Triplet Network

Grad CAM Activation maps

Face-forensics

FaceForensics++ dataset contains four types of forgeries:

• Face2Face
• FaceSwap
• Deepfakes
• Neural Texture

It contains 1000 manipulated videos of each type and 1000 real videos on which these 4 manipulations have been done.

Final Architecture

TSNE plot of FaceForensics++ dataset

Results

Citation

If you find this work useful, please consider citing the following paper:

@inproceedings{Kumar2020DetectingDW,
 title={Detecting Deepfakes with Metric Learning},
 author={Akash Kumar and Arnav Bhavsar},
 year={2020}
}

Notes

For FaceForensics++ and Celeb-DF dataset, contact the authors of the dataset. The dataset can't be shared with the third party. You need to accept the terms on their pages. Then, they will provide you with the access.

I'm styling codes so that it's easy reproducible to all. If any errors you face in the repo, please raise a issue. (Any place where I should explain more) I'll be happy to resolve it as soon as possible.

Currently updating the repo