Code and documentation for research and development project with a topic in Deep Neural Network Compression as partial fulfillment in Masters of Autonomous Systems program.
Comparison of compression methods in Deep Learning for image classification task. Comparison is done in terms of speed using the backbone of MLMark benchmark. Compression methods observed are as follows:
Dataset used for comparison is CIFAR-10 to mimic real-life situations.
Dataset are processed using the network of ResNet-56 and ResNet-110 with pre-activations. In model distillation mode, both of the network act as a teacher which knowledge are transferred to student networks; ResNet-1, ResNet-10, and ResNet-20
- L1 norm pruning: cloned from https://github.com/Eric-mingjie/rethinking-network-pruning/tree/master/cifar/l1-norm-pruning with minor modifications. Based on the implementation of the paper Pruning Filters For Efficient ConvNets
- Weight level pruning : cloned from https://github.com/Eric-mingjie/rethinking-network-pruning/tree/master/cifar/weight-level with minor modifications. Based on the implementation of the paper Learning both Weights and Connections for Efficient Neural Networks
- Knowledge Distillation methods :
- Cloned from https://github.com/peterliht/knowledge-distillation-pytorch with minor modifications. Based on the implementation of the paper Distilling the Knowledge in a Neural Network
- FitNets implementation. Cloned from https://github.com/AberHu/Knowledge-Distillation-Zoo with modifications. Based on the implementation of the paper FitNets: Hints for Thin Deep Nets
- Low Rank Approximations Caffe: Convolutional Architecture for Fast Feature Embedding
- Quantizations (https://github.com/eladhoffer/convNet.pytorch/blob/master/models/modules/quantize.py)
- Results presentations