Comparing unimodal and multimodal emotion classification systems on cohesive data

Master's Thesis by Jakob Kruse

This repository serves as a framework for emotion classification. It implements several classifiers for different data types/modalities like face images, text/sentences, voice/speech, physiological signals (heartrate, acceleration from a smartwatch) and more sensors. The classifiers can be trained and then used for inference. For training, recommended datasets and how to set them up is described in data/DESCRIPTION.md.

Emotions: All classifiers used in this work use 7 emotion classes. The classes are the six basic emotions defined by Paul Ekman: happiness, sadness, anger, fear, surprise, disgust; and an additional neutral class. Existing emotion classification models have been retrained on these seven classes in order to make the models comparable and have a single interface for all models.

Classifiers: All classifiers use the same interface for emotion classification.

⚙️ Setup

I recommend setting up a new virtual environment in the venv folder.

How to set up a virtual environment in Python 3

sudo apt install python3-pip python3-venv
python -m venv venv
source venv/bin/activate

Then, install the required packages:

pip install -r requirements.txt

To download the datasets, please refer to data/DESCRIPTION.md. Some of the datasets are publicly available, to download them you can use:

bash data/prepare_all.sh

Prepare all the classification models:

bash models/install_all.sh

Windows Installation Instructions

All installation scripts are made for Linux and don't work out of the box for Windows. These steps are required (from root folder):

• cd .\models\speech; git clone https://github.com/GasserElbanna/serab-byols.git; python -m pip install -e ./serab-byols
• python -m textblob.download_corpora
• In the file emotion-recognition\models\speech\serab-byols\byol_a\common.py in line 20, change the sound backend from "sox_io" to "soundfile"
• To download the datasets, manual steps are required, as the download scripts are not made for Windows.

⬇️ Download models

To download the pretrained models described in my thesis, use the script:

bash models/download_model.sh --data <modality> --model <model_name>

Currently, only the best model for each modality is available. For details on the available options for --data and --model, please refer to the help:

bash models/download_model.sh -h

🧪 Testing

Run the tests using:

python -m pytest tests

Run the tests with coverage:

python -m pytest --cov=src/ tests/

🔮 Overview of the files

📂 .github	Folder containing Github CI configurations.
📂 data	Folder that contains all the training and evaluation datasets.
📂 docs	Folder that contains the documentation wiht sphinx.
📂 experiments	Folder with experiment configurations and scripts for training models.
📂 models	Folder that contains trained models for emotion classification.
📂 src	Folder with source code for this project.
📃 .gitignore	Files that are ignored by git.
📃 .pre-commit-config.yaml	Linter configuration file.
📃 LICENSE	MIT License file.
📃 pyproject.toml	Linter and Test configurations.
📃 README.md	Explanation (You are here).
📃 requirements.txt	Python Package requirements for the project.
📃 setup.cfg	Another linter configuration file.

✒️ Linters

Install the pre-commit hooks for linting:

pre-commit install

To run all linters manually, use:

pre-commit

Note: only changes added to git are included in linting when using the pre-commit command.

You can also run the single linters one at a time to apply the linter to unstaged files:

black --check .
flake8 .
isort --check-only .