[ModuleSparsificationInfo] Add `logabble_items` method #1468

dbogunowicz · 2023-03-22T17:18:56Z

Feature Preview

from sparseml.pytorch.models import ModelRegistry
from sparseml.pytorch.utils import PythonLogger, LoggerManager, log_module_sparsification_info
from sparseml.pytorch.optim import ScheduledModifierManager
import logging

logging.basicConfig(level=logging.DEBUG)

ZOO_STUB = "zoo:cv/classification/resnet_v1-18/pytorch/sparseml/imagenet/pruned-conservative"
QUANT_RECIPE = """
!QuantizationModifier
    start_epoch: 0.0
    scheme:
        input_activations:
            num_bits: 8
            symmetric: False
        weights:
            num_bits: 8
            symmetric: True
        scheme_overrides:
            feature_extractor: "default"
            classifier:
                input_activations:
                    num_bits: 8
                    symmetric: False
                weights: null
            Conv2d:
                input_activations:
                    num_bits: 8
                    symmetric: True
        ignore: ["ReLU", "input"]
        disable_quantization_observer_epoch: 2.0
        freeze_bn_stats_epoch: 3.0
        model_fuse_fn_name: 'fuse_module'
        strict: True"""

# SparseZoo stub to pre-trained sparse-quantized ResNet-50 for imagenet dataset
model = ModelRegistry.create(
    key="resnet18",
    pretrained_path=ZOO_STUB,
)

manager = ScheduledModifierManager.from_yaml(QUANT_RECIPE)
manager.apply(model)

log_module_sparsification_info(model, logger = PythonLogger())

Returns logs:

Note: every log has three copies returned because of the bug, fix here: #1483

python SparsificationSummaries/OperationCounts step None: {'ConvBnReLU2d': 9, 'MaxPool2d': 1, 'ConvBn2d': 11, 'AdaptiveAvgPool2d': 1, 'Linear': 1, 'Softmax': 1}
python SparsificationSummaries/OperationCounts step None: {'ConvBnReLU2d': 9, 'MaxPool2d': 1, 'ConvBn2d': 11, 'AdaptiveAvgPool2d': 1, 'Linear': 1, 'Softmax': 1}
python SparsificationSummaries/OperationCounts step None: {'ConvBnReLU2d': 9, 'MaxPool2d': 1, 'ConvBn2d': 11, 'AdaptiveAvgPool2d': 1, 'Linear': 1, 'Softmax': 1}
python SparsificationSummaries/OperationCounts step None: {'ConvBnReLU2d': 9, 'MaxPool2d': 1, 'ConvBn2d': 11, 'AdaptiveAvgPool2d': 1, 'Linear': 1, 'Softmax': 1}
python SparsificationSummaries/ParameterCounts step None: {'input.conv.module.weight': 9408, 'input.conv.module.bn.weight': 64, 'input.conv.module.bn.bias': 64, 'sections.0.0.conv1.module.weight': 36864, 'sections.0.0.conv1.module.bn.weight': 64, 'sections.0.0.conv1.module.bn.bias': 64, 'sections.0.0.conv2.module.weight': 36864, 'sections.0.0.conv2.module.bn.weight': 64, 'sections.0.0.conv2.module.bn.bias': 64, 'sections.0.1.conv1.module.weight': 36864, 'sections.0.1.conv1.module.bn.weight': 64, 'sections.0.1.conv1.module.bn.bias': 64, 'sections.0.1.conv2.module.weight': 36864, 'sections.0.1.conv2.module.bn.weight': 64, 'sections.0.1.conv2.module.bn.bias': 64, 'sections.1.0.conv1.module.weight': 73728, 'sections.1.0.conv1.module.bn.weight': 128, 'sections.1.0.conv1.module.bn.bias': 128, 'sections.1.0.conv2.module.weight': 147456, 'sections.1.0.conv2.module.bn.weight': 128, 'sections.1.0.conv2.module.bn.bias': 128, 'sections.1.0.identity.conv.module.weight': 8192, 'sections.1.0.identity.conv.module.bn.weight': 128, 'sections.1.0.identity.conv.module.bn.bias': 128, 'sections.1.1.conv1.module.weight': 147456, 'sections.1.1.conv1.module.bn.weight': 128, 'sections.1.1.conv1.module.bn.bias': 128, 'sections.1.1.conv2.module.weight': 147456, 'sections.1.1.conv2.module.bn.weight': 128, 'sections.1.1.conv2.module.bn.bias': 128, 'sections.2.0.conv1.module.weight': 294912, 'sections.2.0.conv1.module.bn.weight': 256, 'sections.2.0.conv1.module.bn.bias': 256, 'sections.2.0.conv2.module.weight': 589824, 'sections.2.0.conv2.module.bn.weight': 256, 'sections.2.0.conv2.module.bn.bias': 256, 'sections.2.0.identity.conv.module.weight': 32768, 'sections.2.0.identity.conv.module.bn.weight': 256, 'sections.2.0.identity.conv.module.bn.bias': 256, 'sections.2.1.conv1.module.weight': 589824, 'sections.2.1.conv1.module.bn.weight': 256, 'sections.2.1.conv1.module.bn.bias': 256, 'sections.2.1.conv2.module.weight': 589824, 'sections.2.1.conv2.module.bn.weight': 256, 'sections.2.1.conv2.module.bn.bias': 256, 'sections.3.0.conv1.module.weight': 1179648, 'sections.3.0.conv1.module.bn.weight': 512, 'sections.3.0.conv1.module.bn.bias': 512, 'sections.3.0.conv2.module.weight': 2359296, 'sections.3.0.conv2.module.bn.weight': 512, 'sections.3.0.conv2.module.bn.bias': 512, 'sections.3.0.identity.conv.module.weight': 131072, 'sections.3.0.identity.conv.module.bn.weight': 512, 'sections.3.0.identity.conv.module.bn.bias': 512, 'sections.3.1.conv1.module.weight': 2359296, 'sections.3.1.conv1.module.bn.weight': 512, 'sections.3.1.conv1.module.bn.bias': 512, 'sections.3.1.conv2.module.weight': 2359296, 'sections.3.1.conv2.module.bn.weight': 512, 'sections.3.1.conv2.module.bn.bias': 512, 'classifier.fc.module.weight': 512000, 'classifier.fc.module.bias': 1000}
...
python SparsificationSummaries/QuantizedOperations/count step None: 24
python SparsificationSummaries/QuantizedOperations/count step None: 24
python SparsificationSummaries/QuantizedOperations/count step None: 24
python SparsificationSummaries/QuantizedOperations/count step None: 24
python SparsificationSummaries/QuantizedOperations/percent step None: 1.0
python SparsificationSummaries/QuantizedOperations/percent step None: 1.0
python SparsificationSummaries/QuantizedOperations/percent step None: 1.0
python SparsificationSummaries/QuantizedOperations/percent step None: 1.0
python SparsificationSummaries/PrunedParameters/count step None: 19
python SparsificationSummaries/PrunedParameters/count step None: 19
python SparsificationSummaries/PrunedParameters/count step None: 19
python SparsificationSummaries/PrunedParameters/count step None: 19
python SparsificationSummaries/PrunedParameters/percent step None: 0.3064516129032258
python SparsificationSummaries/PrunedParameters/percent step None: 0.3064516129032258
python SparsificationSummaries/PrunedParameters/percent step None: 0.3064516129032258
python SparsificationSummaries/PrunedParameters/percent step None: 0.3064516129032258
python SparsificationPruning/ZeroParameters/input.conv.module.weight/count step None: 0
python SparsificationPruning/ZeroParameters/input.conv.module.weight/count step None: 0
python SparsificationPruning/ZeroParameters/input.conv.module.weight/count step None: 0
python SparsificationPruning/ZeroParameters/input.conv.module.weight/count step None: 0
python SparsificationPruning/ZeroParameters/input.conv.module.weight/percent step None: 0.0
python SparsificationPruning/ZeroParameters/input.conv.module.weight/percent step None: 0.0
python SparsificationPruning/ZeroParameters/input.conv.module.weight/percent step None: 0.0
python SparsificationPruning/ZeroParameters/input.conv.module.weight/percent step None: 0.0
python SparsificationPruning/ZeroParameters/input.conv.module.bn.weight/count step None: 0
python SparsificationPruning/ZeroParameters/input.conv.module.bn.weight/count step None: 0
python SparsificationPruning/ZeroParameters/input.conv.module.bn.weight/count step None: 0
python SparsificationPruning/ZeroParameters/input.conv.module.bn.weight/count step None: 0
python SparsificationPruning/ZeroParameters/input.conv.module.bn.weight/percent step None: 0.0
python SparsificationPruning/ZeroParameters/input.conv.module.bn.weight/percent step None: 0.0
python SparsificationPruning/ZeroParameters/input.conv.module.bn.weight/percent step None: 0.0
python SparsificationPruning/ZeroParameters/input.conv.module.bn.weight/percent step None: 0.0
...
python SparsificationQuantization/ConvBnReLU2d/enabled step None: True
python SparsificationQuantization/ConvBnReLU2d/enabled step None: True
python SparsificationQuantization/ConvBnReLU2d/enabled step None: True
python SparsificationQuantization/ConvBnReLU2d/enabled step None: True
python SparsificationQuantization/ConvBnReLU2d/precision/weights/num_bits step None: 8
python SparsificationQuantization/ConvBnReLU2d/precision/weights/num_bits step None: 8
python SparsificationQuantization/ConvBnReLU2d/precision/weights/num_bits step None: 8
python SparsificationQuantization/ConvBnReLU2d/precision/weights/num_bits step None: 8
python SparsificationQuantization/ConvBnReLU2d/precision/input_activations/num_bits step None: 8
python SparsificationQuantization/ConvBnReLU2d/precision/input_activations/num_bits step None: 8
python SparsificationQuantization/ConvBnReLU2d/precision/input_activations/num_bits step None: 8
python SparsificationQuantization/ConvBnReLU2d/precision/input_activations/num_bits step None: 8
python SparsificationQuantization/MaxPool2d/enabled step None: True
python SparsificationQuantization/MaxPool2d/enabled step None: True
python SparsificationQuantization/MaxPool2d/enabled step None: True
python SparsificationQuantization/MaxPool2d/enabled step None: True
python SparsificationQuantization/MaxPool2d/precision/weights/num_bits step None: 8
python SparsificationQuantization/MaxPool2d/precision/weights/num_bits step None: 8
python SparsificationQuantization/MaxPool2d/precision/weights/num_bits step None: 8
python SparsificationQuantization/MaxPool2d/precision/weights/num_bits step None: 8
python SparsificationQuantization/MaxPool2d/precision/input_activations/num_bits step None: 8
python SparsificationQuantization/MaxPool2d/precision/input_activations/num_bits step None: 8
python SparsificationQuantization/MaxPool2d/precision/input_activations/num_bits step None: 8
python SparsificationQuantization/MaxPool2d/precision/input_activations/num_bits step None: 8
python SparsificationQuantization/ConvBnReLU2d_1/enabled step None: True
python SparsificationQuantization/ConvBnReLU2d_1/enabled step None: True
python SparsificationQuantization/ConvBnReLU2d_1/enabled step None: True
python SparsificationQuantization/ConvBnReLU2d_1/enabled step None: True
...

dbogunowicz · 2023-03-22T17:20:37Z

src/sparseml/pytorch/utils/sparsification_info/module_sparsification_info.py

@@ -58,5 +59,13 @@ def from_module(cls, module: torch.nn.Module) -> "ModuleSparsificationInfo":
            quantization_info=SparsificationQuantization.from_module(module),
        )

-    def loggable_items(self) -> Iterable[Tuple[str, float]]:
-        raise NotImplementedError()
+    def loggable_items(self) -> Generator[Tuple[str, Any], None, None]:


Thoughts on making it an __iter__, such that we can do:

info = ModuleSparsificationInfo.from_module(...) for tag, item in info: ....

let's not do that for now since the logged items here should directly match with the logging targets from our PRD and not actually be representative of what's contained in this class

dbogunowicz · 2023-03-22T17:23:15Z

Also note: while working on that feature in noticed a bug in quantization info. Because operators will have the same name (Conv2D for any of the convolutional operators), we need to append some identifier (like an integer) to make them separate from each other. We need all of them to store information e.g. about their dtype or whether quantization is enabled.

bfineran · 2023-03-22T22:57:55Z

src/sparseml/pytorch/utils/sparsification_info/module_sparsification_info.py

@@ -58,5 +59,13 @@ def from_module(cls, module: torch.nn.Module) -> "ModuleSparsificationInfo":
            quantization_info=SparsificationQuantization.from_module(module),
        )

-    def loggable_items(self) -> Iterable[Tuple[str, float]]:
-        raise NotImplementedError()
+    def loggable_items(self) -> Generator[Tuple[str, Any], None, None]:


let's not do that for now since the logged items here should directly match with the logging targets from our PRD and not actually be representative of what's contained in this class

src/sparseml/pytorch/utils/sparsification_info/helpers.py

Co-authored-by: Benjamin Fineran <bfineran@users.noreply.github.com>

…into feature/damian/loggable_items

src/sparseml/pytorch/utils/sparsification_info/helpers.py

…ication' into feature/damian/loggable_items

…tion_info' into feature/damian/loggable_items

corey-nm · 2023-03-27T16:25:29Z

src/sparseml/pytorch/utils/sparsification_info/configs.py

@@ -108,6 +108,22 @@ def from_module(
            operation_counts=Counter([op.__class__.__name__ for op in operations]),
        )

+    def loggable_items(


Thoughts on making an abstract class that all these inherit from? That's the only other thing I can think of for this for ensuring the object you get has loggable_items (or you could just call hasattr)

Yeah, sounds like a good idea.

corey-nm

lgtm, nice use of generators

bfineran

LGTM pending comment

bfineran · 2023-03-27T17:44:31Z

src/sparseml/pytorch/utils/sparsification_info/configs.py

+
+            quantization_scheme = self.quantization_scheme[operation]
+            if quantization_scheme is None:
+                yield f"{main_tag}/{operation}/precision", None


Why None instead of 32/16 based on the param?

* initial_commit * [ModuleSparsificationInfo] Proposal of the main logic (#1454) * initial commit * initial developement * sync with ben * prototype ready * included ben's comments * Update src/sparseml/pytorch/utils/sparsification_info/configs.py Co-authored-by: Benjamin Fineran <bfineran@users.noreply.github.com> * Update src/sparseml/pytorch/utils/sparsification_info/configs.py Co-authored-by: Benjamin Fineran <bfineran@users.noreply.github.com> * address comments * fix quantization logic * remove identities from leaf operations * fix the Identity removal * [ModuleSparsificationInfo][Tests] Proposal of the main logic (#1479) * Update src/sparseml/pytorch/utils/sparsification_info/helpers.py Co-authored-by: Benjamin Fineran <bfineran@users.noreply.github.com> * addressing PR comments --------- Co-authored-by: Benjamin Fineran <bfineran@users.noreply.github.com> * [ModuleSparsificationInfo] Add `logabble_items` method (#1468) * initial commit * initial developement * sync with ben * prototype ready * included ben's comments * initial commit * Update src/sparseml/pytorch/utils/sparsification_info/configs.py Co-authored-by: Benjamin Fineran <bfineran@users.noreply.github.com> * Update src/sparseml/pytorch/utils/sparsification_info/configs.py Co-authored-by: Benjamin Fineran <bfineran@users.noreply.github.com> * address comments * fix quantization logic * formatting standardised to the PRD * remove identities from leaf operations * fix the Identity removal * initial commit * cleanup * correct tests * address PR comments --------- Co-authored-by: Benjamin Fineran <bfineran@users.noreply.github.com> * [ModuleSparisificationInfo] LoggingModifier (#1484) * initial commit * initial developement * sync with ben * prototype ready * included ben's comments * initial commit * Update src/sparseml/pytorch/utils/sparsification_info/configs.py Co-authored-by: Benjamin Fineran <bfineran@users.noreply.github.com> * Update src/sparseml/pytorch/utils/sparsification_info/configs.py Co-authored-by: Benjamin Fineran <bfineran@users.noreply.github.com> * address comments * fix quantization logic * formatting standardised to the PRD * remove identities from leaf operations * fix the Identity removal * initial commit * initial commit * checkpoint * Delete modifier_logging.py * Apply suggestions from code review * tested the modifier * Apply suggestions from code review --------- Co-authored-by: Benjamin Fineran <bfineran@users.noreply.github.com> * [ModuleSparsificationInfo][Tests] SparsificationLoggingModifier (#1485) * Apply suggestions from code review * Trigger tests --------- Co-authored-by: Benjamin Fineran <bfineran@users.noreply.github.com>

dbogunowicz added 6 commits March 17, 2023 15:00

initial commit

218dd8a

initial developement

b21bbca

sync with ben

da95fcf

prototype ready

c10ebfa

included ben's comments

fe2ce6f

initial commit

5e633f4

dbogunowicz commented Mar 22, 2023

View reviewed changes

dbogunowicz changed the title ~~[Sparsification Info] Add logabble_items method~~ [ModuleSparsificationInfo] Add logabble_items method Mar 22, 2023

bfineran reviewed Mar 22, 2023

View reviewed changes

dbogunowicz and others added 6 commits March 23, 2023 12:05

Update src/sparseml/pytorch/utils/sparsification_info/configs.py

bff507d

Co-authored-by: Benjamin Fineran <bfineran@users.noreply.github.com>

Update src/sparseml/pytorch/utils/sparsification_info/configs.py

84884b1

Co-authored-by: Benjamin Fineran <bfineran@users.noreply.github.com>

address comments

a25fd82

fix quantization logic

fea32c4

Merge remote-tracking branch 'origin/feature/damian/summary_configs' …

e24650c

…into feature/damian/loggable_items

formatting standardised to the PRD

4289bda

bfineran suggested changes Mar 24, 2023

View reviewed changes

src/sparseml/pytorch/utils/sparsification_info/helpers.py Show resolved Hide resolved

dbogunowicz added 4 commits March 24, 2023 15:16

remove identities from leaf operations

c9f708d

fix the Identity removal

68f0bb4

initial commit

b005900

enforce new structure + add tests

a053c44

dbogunowicz changed the base branch from feature/damian/summary_configs to feature/damian/add_tests_sparsification March 24, 2023 19:04

Merge remote-tracking branch 'origin/feature/damian/add_tests_sparsif…

5c618c1

…ication' into feature/damian/loggable_items

Base automatically changed from feature/damian/add_tests_sparsification to feature/damian/summary_configs March 27, 2023 15:28

Base automatically changed from feature/damian/summary_configs to feature/damian/module_sparsification_info March 27, 2023 16:00

dbogunowicz added 3 commits March 27, 2023 16:10

Merge remote-tracking branch 'origin/feature/damian/module_sparsifica…

37fb9b3

…tion_info' into feature/damian/loggable_items

cleanup

896935a

correct tests

e050328

dbogunowicz requested review from bfineran and corey-nm March 27, 2023 16:23

corey-nm reviewed Mar 27, 2023

View reviewed changes

corey-nm approved these changes Mar 27, 2023

View reviewed changes

bfineran approved these changes Mar 27, 2023

View reviewed changes

address PR comments

048b8ea

dbogunowicz merged commit 8052c54 into feature/damian/module_sparsification_info Mar 28, 2023

dbogunowicz deleted the feature/damian/loggable_items branch March 28, 2023 09:12

dbogunowicz mentioned this pull request Mar 28, 2023

SparsificationLoggingModifier implementation #1453

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ModuleSparsificationInfo] Add `logabble_items` method #1468

[ModuleSparsificationInfo] Add `logabble_items` method #1468

dbogunowicz commented Mar 22, 2023 •

edited

Loading

dbogunowicz Mar 22, 2023

bfineran Mar 22, 2023

dbogunowicz commented Mar 22, 2023

bfineran Mar 22, 2023

corey-nm Mar 27, 2023

dbogunowicz Mar 27, 2023

corey-nm left a comment

bfineran left a comment

bfineran Mar 27, 2023

[ModuleSparsificationInfo] Add logabble_items method #1468

[ModuleSparsificationInfo] Add logabble_items method #1468

Conversation

dbogunowicz commented Mar 22, 2023 • edited Loading

Feature Preview

dbogunowicz Mar 22, 2023

Choose a reason for hiding this comment

bfineran Mar 22, 2023

Choose a reason for hiding this comment

dbogunowicz commented Mar 22, 2023

bfineran Mar 22, 2023

Choose a reason for hiding this comment

corey-nm Mar 27, 2023

Choose a reason for hiding this comment

dbogunowicz Mar 27, 2023

Choose a reason for hiding this comment

corey-nm left a comment

Choose a reason for hiding this comment

bfineran left a comment

Choose a reason for hiding this comment

bfineran Mar 27, 2023

Choose a reason for hiding this comment

[ModuleSparsificationInfo] Add `logabble_items` method #1468

[ModuleSparsificationInfo] Add `logabble_items` method #1468

dbogunowicz commented Mar 22, 2023 •

edited

Loading