- Python 3 (>=3.7)
- Pipenv
- CUDA-compatible GPU (for training)
As long as you have Python3 and Pipenv installed, run the following command:
$ pipenv installPipenv will automatically create and install all the required dependencies for the project.
The quickest way to train some of the models is to load one of the notebooks in Google Colab.
For example, running the notebook found in notebooks/kd-mnist.ipynb you can train both the Teacher and Student
models on the full MNIST dataset as well as performing KD from the Teacher to the Student.
The Notebook outputs the accuracy of the models.
After running the notebook, you could then load the created model into the notebook notebook/kd-analysis.ipynb
which allows you to generate graphs to see how temperature affects the logits of the models.
Otherwise, you can perform training using the provided run.py script which uses the code from the notebooks and train.py
to train the models from scratch and perform KD using a varitey of hyperparameters.
First, alter the run.py script to run your desired experiment, then:
$ pipenv shell
$ python notebooks/run.pyAlternatively, you can launch Jupyter Lab locally (the package is included in the Pipfile).
$ pipenv shell
$ jupyter lab/modelscontains the best trained models for the Teacher, Student and Assistant networks/notebookscontains all the code for the project, it is broken down as follows:kd-analysisallows you to generate plots based on the pickled data produced during trainingkd-assistantcontains the code from the TAKD experimentskd-cifarcontains all the experiments for KD teacher to student using the whole dataset and only 3% of the datakd-mnistsame as above, except for it uses the MNIST datasetmodel.pycontains network definitions and helper create functionstrain.pycontains helper functions for training the networksutils.pycontains functions for loading datasets and calculating model accuracyrun.pycontains helper functions for running different experiments