SparsificationLoggingModifier implementation

[dbogunowicz]

Feature Description

Adds SparsificationLoggingModifier. It can be used in the sparsification recipe to periodically log the information about the sparsification properties of the model (e.g. level of pruning, the precision of parameters, etc).

Feature Preview

sparseml.image_classification.train --dataset imagenette --arch_key mobilenet --recipe-path recipe.yaml --dataset_path . --train_batch_size 64  --test_batch_size 64

where recipe.yaml is:

version: 1.1.0

training_modifiers:
  - !EpochRangeModifier
    start_epoch: 0
    end_epoch: 4.0

  - !SparsificationLoggingModifier
    start_epoch: 0.0
    end_epoch: 4.0
    update_frequency: 2

Results in:

# starting logging
Logging all SparseML modifier-level logs to sparse_logs/28-03-2023_10.31.51.log
# test pass at epoch 0
python Test/__loss__ Summary [-1 - 1679999514.1273096]: 2.29128360748291
...
python Test/Minutes per epoch [-1 - 1679999514.1289337]: 0.03270106315612793
# logging sparsity information at epoch 0
python SparsificationSummaries/OperationCounts [0.0 - 1679999520.515994]: {'Conv2d': 27, 'BatchNorm2d': 27, 'ReLU': 27, 'AvgPool2d': 1, 'Linear': 1, 'Softmax': 1}
python SparsificationSummaries/ParameterCounts [0.0 - 1679999520.5214868]: {'input.conv.weight': 864, 'input.bn.weight': 32, 'input.bn.bias': 32, 'sections.0.0.depth.conv.weight': 288, 'sections.0.0.depth.bn.weight': 32, 'sections.0.0.depth.bn.bias': 32, 'sections.0.0.point.conv.weight': 2048, 'sections.0.0.point.bn.weight': 64, 'sections.0.0.point.bn.bias': 64, 'sections.1.0.depth.conv.weight': 576, 'sections.1.0.depth.bn.weight': 64, 'sections.1.0.depth.bn.bias': 64, 'sections.1.0.point.conv.weight': 8192, 'sections.1.0.point.bn.weight': 128, 'sections.1.0.point.bn.bias': 128, 'sections.1.1.depth.conv.weight': 1152, 'sections.1.1.depth.bn.weight': 128, 'sections.1.1.depth.bn.bias': 128, 'sections.1.1.point.conv.weight': 16384, 'sections.1.1.point.bn.weight': 128, 'sections.1.1.point.bn.bias': 128, 'sections.2.0.depth.conv.weight': 1152, 'sections.2.0.depth.bn.weight': 128, 'sections.2.0.depth.bn.bias': 128, 'sections.2.0.point.conv.weight': 32768, 'sections.2.0.point.bn.weight': 256, 'sections.2.0.point.bn.bias': 256, 'sections.2.1.depth.conv.weight': 2304, 'sections.2.1.depth.bn.weight': 256, 'sections.2.1.depth.bn.bias': 256, 'sections.2.1.point.conv.weight': 65536, 'sections.2.1.point.bn.weight': 256, 'sections.2.1.point.bn.bias': 256, 'sections.3.0.depth.conv.weight': 2304, 'sections.3.0.depth.bn.weight': 256, 'sections.3.0.depth.bn.bias': 256, 'sections.3.0.point.conv.weight': 131072, 'sections.3.0.point.bn.weight': 512, 'sections.3.0.point.bn.bias': 512, 'sections.3.1.depth.conv.weight': 4608, 'sections.3.1.depth.bn.weight': 512, 'sections.3.1.depth.bn.bias': 512, 'sections.3.1.point.conv.weight': 262144, 'sections.3.1.point.bn.weight': 512, 'sections.3.1.point.bn.bias': 512, 'sections.3.2.depth.conv.weight': 4608, 'sections.3.2.depth.bn.weight': 512, 'sections.3.2.depth.bn.bias': 512, 'sections.3.2.point.conv.weight': 262144, 'sections.3.2.point.bn.weight': 512, 'sections.3.2.point.bn.bias': 512, 'sections.3.3.depth.conv.weight': 4608, 'sections.3.3.depth.bn.weight': 512, 'sections.3.3.depth.bn.bias': 512, 'sections.3.3.point.conv.weight': 262144, 'sections.3.3.point.bn.weight': 512, 'sections.3.3.point.bn.bias': 512, 'sections.3.4.depth.conv.weight': 4608, 'sections.3.4.depth.bn.weight': 512, 'sections.3.4.depth.bn.bias': 512, 'sections.3.4.point.conv.weight': 262144, 'sections.3.4.point.bn.weight': 512, 'sections.3.4.point.bn.bias': 512, 'sections.3.5.depth.conv.weight': 4608, 'sections.3.5.depth.bn.weight': 512, 'sections.3.5.depth.bn.bias': 512, 'sections.3.5.point.conv.weight': 262144, 'sections.3.5.point.bn.weight': 512, 'sections.3.5.point.bn.bias': 512, 'sections.4.0.depth.conv.weight': 4608, 'sections.4.0.depth.bn.weight': 512, 'sections.4.0.depth.bn.bias': 512, 'sections.4.0.point.conv.weight': 524288, 'sections.4.0.point.bn.weight': 1024, 'sections.4.0.point.bn.bias': 1024, 'sections.4.1.depth.conv.weight': 9216, 'sections.4.1.depth.bn.weight': 1024, 'sections.4.1.depth.bn.bias': 1024, 'sections.4.1.point.conv.weight': 1048576, 'sections.4.1.point.bn.weight': 1024, 'sections.4.1.point.bn.bias': 1024, 'classifier.fc.weight': 10240, 'classifier.fc.bias': 10}
python SparsificationSummaries/QuantizedOperations/count [0.0 - 1679999520.5881698]: 0
python SparsificationSummaries/QuantizedOperations/percent [0.0 - 1679999520.5883384]: 0.0
...
python SparsificationQuantization/Linear/enabled [0.0 - 1679999520.64014]: False
python SparsificationQuantization/Linear/precision.weights/num_bits [0.0 - 1679999520.6402037]: 32
python SparsificationQuantization/Softmax/enabled [0.0 - 1679999520.6402755]: False
python SparsificationQuantization/Softmax/precision [0.0 - 1679999520.6403413]: None
# train pass at epoch 0
python Train/Learning Rate [0 - 1679999520.6444604]: 1e-09
...
python Train/Learning Rate Summary [0 - 1679999533.215367]: 1e-09
# test pass at epoch 0
python Test/__loss__ Summary [0 - 1679999534.5384443]: 2.2875683307647705
...
python Test/Minutes per epoch [0 - 1679999534.5399299]: 0.016155858834584553
# train pass at epoch 1
python Train/Learning Rate [202 - 1679999535.5298498]: 1e-09
python Train/__loss__ [202 - 1679999535.5331175]: 2.295095682144165
...
python Train/Minutes per epoch [1 - 1679999548.0556235]: 0.2193891167640686
python Train/Learning Rate Summary [1 - 1679999548.0557492]: 1e-09
# test pass at epoch 1
python Test/__loss__ Summary [1 - 1679999549.3739681]: 2.2884953022003174
...
python Test/Minutes per epoch [1 - 1679999549.375715]: 0.016250451405843098
# logging sparsity information at epoch 2
python SparsificationSummaries/OperationCounts [2.0 - 1679999550.4210393]: {'Conv2d': 27, 'BatchNorm2d': 27, 'ReLU': 27, 'AvgPool2d': 1, 'Linear': 1, 'Softmax': 1}
python SparsificationSummaries/ParameterCounts [2.0 - 1679999550.4223588]: {'input.conv.weight': 864, 'input.bn.weight': 32, 'input.bn.bias': 32, 'sections.0.0.depth.conv.weight': 288, 'sections.0.0.depth.bn.weight': 32, 'sections.0.0.depth.bn.bias': 32, 'sections.0.0.point.conv.weight': 2048, 'sections.0.0.point.bn.weight': 64, 'sections.0.0.point.bn.bias': 64, 'sections.1.0.depth.conv.weight': 576, 'sections.1.0.depth.bn.weight': 64, 'sections.1.0.depth.bn.bias': 64, 'sections.1.0.point.conv.weight': 8192, 'sections.1.0.point.bn.weight': 128, 'sections.1.0.point.bn.bias': 128, 'sections.1.1.depth.conv.weight': 1152, 'sections.1.1.depth.bn.weight': 128, 'sections.1.1.depth.bn.bias': 128, 'sections.1.1.point.conv.weight': 16384, 'sections.1.1.point.bn.weight': 128, 'sections.1.1.point.bn.bias': 128, 'sections.2.0.depth.conv.weight': 1152, 'sections.2.0.depth.bn.weight': 128, 'sections.2.0.depth.bn.bias': 128, 'sections.2.0.point.conv.weight': 32768, 'sections.2.0.point.bn.weight': 256, 'sections.2.0.point.bn.bias': 256, 'sections.2.1.depth.conv.weight': 2304, 'sections.2.1.depth.bn.weight': 256, 'sections.2.1.depth.bn.bias': 256, 'sections.2.1.point.conv.weight': 65536, 'sections.2.1.point.bn.weight': 256, 'sections.2.1.point.bn.bias': 256, 'sections.3.0.depth.conv.weight': 2304, 'sections.3.0.depth.bn.weight': 256, 'sections.3.0.depth.bn.bias': 256, 'sections.3.0.point.conv.weight': 131072, 'sections.3.0.point.bn.weight': 512, 'sections.3.0.point.bn.bias': 512, 'sections.3.1.depth.conv.weight': 4608, 'sections.3.1.depth.bn.weight': 512, 'sections.3.1.depth.bn.bias': 512, 'sections.3.1.point.conv.weight': 262144, 'sections.3.1.point.bn.weight': 512, 'sections.3.1.point.bn.bias': 512, 'sections.3.2.depth.conv.weight': 4608, 'sections.3.2.depth.bn.weight': 512, 'sections.3.2.depth.bn.bias': 512, 'sections.3.2.point.conv.weight': 262144, 'sections.3.2.point.bn.weight': 512, 'sections.3.2.point.bn.bias': 512, 'sections.3.3.depth.conv.weight': 4608, 'sections.3.3.depth.bn.weight': 512, 'sections.3.3.depth.bn.bias': 512, 'sections.3.3.point.conv.weight': 262144, 'sections.3.3.point.bn.weight': 512, 'sections.3.3.point.bn.bias': 512, 'sections.3.4.depth.conv.weight': 4608, 'sections.3.4.depth.bn.weight': 512, 'sections.3.4.depth.bn.bias': 512, 'sections.3.4.point.conv.weight': 262144, 'sections.3.4.point.bn.weight': 512, 'sections.3.4.point.bn.bias': 512, 'sections.3.5.depth.conv.weight': 4608, 'sections.3.5.depth.bn.weight': 512, 'sections.3.5.depth.bn.bias': 512, 'sections.3.5.point.conv.weight': 262144, 'sections.3.5.point.bn.weight': 512, 'sections.3.5.point.bn.bias': 512, 'sections.4.0.depth.conv.weight': 4608, 'sections.4.0.depth.bn.weight': 512, 'sections.4.0.depth.bn.bias': 512, 'sections.4.0.point.conv.weight': 524288, 'sections.4.0.point.bn.weight': 1024, 'sections.4.0.point.bn.bias': 1024, 'sections.4.1.depth.conv.weight': 9216, 'sections.4.1.depth.bn.weight': 1024, 'sections.4.1.depth.bn.bias': 1024, 'sections.4.1.point.conv.weight': 1048576, 'sections.4.1.point.bn.weight': 1024, 'sections.4.1.point.bn.bias': 1024, 'classifier.fc.weight': 10240, 'classifier.fc.bias': 10}
python SparsificationSummaries/QuantizedOperations/count [2.0 - 1679999550.4265969]: 0
python SparsificationSummaries/QuantizedOperations/percent [2.0 - 1679999550.4267216]: 0.0
...
python SparsificationQuantization/Linear/enabled [2.0 - 1679999550.515244]: False
python SparsificationQuantization/Linear/precision.weights/num_bits [2.0 - 1679999550.5153124]: 32
python SparsificationQuantization/Softmax/enabled [2.0 - 1679999550.515388]: False
python SparsificationQuantization/Softmax/precision [2.0 - 1679999550.5154629]: None
# train pass at epoch 2
python Train/Learning Rate [404 - 1679999550.5190117]: 1e-09
python Train/__loss__ [404 - 1679999550.5193589]: 2.2715582847595215
...
python Train/Minutes per epoch [2 - 1679999563.0662048]: 0.22129076719284058
python Train/Learning Rate Summary [2 - 1679999563.0663579]: 1e-09
# test pass at epoch 2
python Test/__loss__ Summary [2 - 1679999564.411749]: 2.2932071685791016
...
python Test/Minutes per epoch [2 - 1679999564.413608]: 0.01665269136428833
# train pass at epoch 3
python Train/Learning Rate [606 - 1679999565.3260324]: 1e-09
...
python Train/Minutes per epoch [3 - 1679999577.9791636]: 0.220275354385376
python Train/Learning Rate Summary [3 - 1679999577.979289]: 1e-09
# test pass at epoch 3
python Test/__loss__ Summary [3 - 1679999579.312438]: 2.291689395904541
...
python Test/Minutes per epoch [3 - 1679999579.3134086]: 0.016451005140940347

Summary

Incorporates the following PRs:

• [ModuleSparsificationInfo][Tests] SparsificationLoggingModifier #1485
• [ModuleSparisificationInfo] LoggingModifier #1484
• [ModuleSparsificationInfo] Add logabble_items method #1468
• [ModuleSparsificationInfo] Proposal of the main logic #1454