This respository contains the code and experiments for the paper Data-Free Model Pruning at Initialization via Expanders, published at the Efficient Deep Learning for Computer Vision CVPR Workshop, 2023.
Authors: James Stewart, Umberto Michieli, and Mete Ozay.
A list of requirements for this project can be found in environment.yml.
To create and activate a conda environment from these requirements, run conda env create -f environment.yml and conda activate config_model_env.
We provide support for fully-connected and sparsified versions of VGG 16 and ResNet (depth 18, 32, 50, 101, and 152). Their implementations can be found in src/model_factory/models.
Implementations of other models should be placed here, and added to src/model_factory/model_factory.py. The list of supported models found in src/utils/config.py should also be updated.
We provide support for the CIFAR-10 and Tiny ImageNet datasets. In the former case, the data will be downloaded to data/; in the latter case, the data should be placed here. The Tiny ImageNet dataset can be downloaded from http://cs231n.stanford.edu/tiny-imagenet-200.zip.
After downloading it, run the script data/prepare_tiny_imagenet.py to prepare the data.
To add a new dataset, a custom data loader should be implemented according to the interface specified by src/data_loader_factory/data_loaders/data_loader.py. The list of supported datasets found in src/utils/config.py should also be updated.
We implement sparse convolutional and linear layers according to three different random graph models: a random graph (Random), a random regular graph (RReg), and a random left-regular graph (XNet). The implementations of these graph masked layers can be found in src/model_factory/layers.
New graph models should be implemented in src/graph_factory/graph_utils.py and added to src/graph_factory/graph_factory.py. The architecture to be sparsified according to the new graph model may then be added to the model factory, as per the above instructions.
After installing the requirements, running the command main.py --model='vgg16bn' --dataset='cifar10' from src/ will train and test a fully-connected VGG 16 model on the CIFAR-10 dataset for 300 epochs using the defauly hyperparameters. We can sparsify this model using random graphs of average degree 3 by running main.py --model='vgg16bn_random' --degree=3 --dataset='cifar10'. For all other options and information on the default hyperparameters, see src/arg_parser/arg_parser.py.
To reproduce the results for VGG 16 and ResNet on both CIFAR-10 and Tiny ImageNet, we provide a number of scripts located in scripts/. From the src/ folder, simply run bash ../scripts/[script name] to execute each of these. The console logs will be written to outputs/.
This work is released under the Attribution-NonCommercial-ShareAlike 4.0 International license. Parts of this work are adapted from PyTorch, which is released under the 3-Clause BSD license.
@InProceedings{stewart2022cvpr,
author = {Stewart, James and Michieli, Umberto and Ozay, Mete},
title = {Data-Free Model Pruning at Initialization via Expanders},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
month = {June},
year = {2023}
}