Facial Emotion Recognition

For better visuals you can watch this video or download this and see locally.

All the models explored are well documented with plots and performance metrics. Here it is.

How to add a new model?

For adding a new model to this repository all you have to do is create the new model, make your data compatible for new model(if required) and then mention the new model, that's it.

for example you want to create a new model named MyModel, then you need to change the following files :-

1. models.py - Within model.py create a new class named MyModel extending BaseModel class. BaseModel is the class which needs to be extended by every model. And then simply implement two methods model_builder and train. All code for building you model goes inside model_builder function. Whereas train function is for training that model.

class MyModel(BaseModel):

   def model_builder(self, **kwargs):
       # build model here
       self.model = my_builded_model
       
   def train(self, **kwargs):
       # train model here

^{Refer to models.py for seeing this process in action.}

2. data_builder.py - Within this file you will make data compatible for your model. For example in your MyModel you are not only feeding the raw face images but also feeding some ROI(region of interest) then you need to crop these ROI for all images and convert them to numpy arrays before feeding into the network. For this within data_builder.py you need to create a new class named ImageToROI extending DataBuilder class. DataBuilder is the class which needs to be extended and you should implement it's build_from_directory method, which returns the data in required format. You can also reuse already made classes if they support your model otherwise you need to create your own.

class ImageToROI(DataBuilder):

    def build_from_directory(self):
        # write code here
        return array
    

^{Refer to data_builder.py for seeing this process in action.}

3. trainer.py - This file is only for training purpose, in this file you should mention your new model.
   ^{Refer to trainer.py for seeing this process in action.}

These 3 are the major files you need to change, apart from this there are some already available lr_scheduler, early_stopping or train_datagen to use, or create your own.

How to train a model?

For training a model all you need to do is run trainer.py with appropriate arguments. There are a variety of arguments which you can mention :-

d - Dataset to train on, fer, feraligned, ck and feraligned+ck are supported.
m - Model to train on, currently cnn and cnn+roi1+roi2 are supported.
em - Emotions to train on, comma separated values, depending on the dataset select any subset from {Happy,Sadness,Surprise,Angry,Fear,Neutral}.
s - 1 to Shuffle before split otherwise 0, default is 1.
rs - Random state to use, default is 42.
tr - Train ratio a value from 0 to 1, default is 0.85.
lrs - Lr scheduler to use, default is None.
es - Early stopping to use, default is None.
tg - Train data generator to use, default is None.
bs - Batch size to use, default is 24.
ep - Max epochs, default is 50.
o - Optimizer to use, adam and nadam are supported, default is adam.
lr - Learning rate to use, default is 0.01.
sa - 1 to save_architecture otherwise 0, default is 0.
sm - 1 to save the model otherwise 0, default is 0.
scm - 1 to save the confusion matrix of test set otherwise 0, default is 0.
sth - 1 to save training history otherwise 0, default is 0.

example :-

python trainer.py -d fer -m MyModel -em Happy,Sadness,Neutral -ep 25 -tr 0.7 -bs 16 -sa 1 -sm 1 -scm 1 -sth 1 -tg 4

This will train MyModel on the fer dataset for 3 classes Happy,Sadness,Neutral, number of epochs are 25, train-ratio is 0.7, batch-size is 16 and we are also saving everything.

All the outputs generated from training will go in their respective folders within the outputs directory. The saved models follows a naming convetion :-
<model name>_<dataset trained on>_<number of emotions trained on>emo

For example in the above model, name would be MyModel_fer_3emo.

How to test a model?

For testing a model run following command

python -m model_testing.test_CNNModel -i webcam -d dnn -m CNNModel_feraligned+ck_5emo

i - Input type, either webcam or path to video file.
d - Detector to use, either dlib or dnn.
m - Model to use, anyone present in the output/models directory.
he - 1 to apply histogram equalization otherwise 0, default is 0.
ws - Window size, comma separated values, default is None.

As each tensorflow model is more than around +50mbs, so adding all the models in this repository will increase it's size drastically hence I added few good performing models.