NNCF2.5 When quantizing the model, an error occurred: "RuntimeError: Could not find the bias value of the node."

[edition3234]

I have an ONNX model that contains convolutional layers but no fully connected layers. Upon inspection with Netron, I found that if a convolutional layer is not directly followed by a BatchNormalization layer, then the convolutional layer has both weights and biases. However, if a convolutional layer is directly followed by a BatchNormalization layer, it only has weights, and the BatchNormalization layer carries the bias. This is the structure of my model. I want to quantize it to int8 using NNCF. Currently, in the get_bias_value function in the nncf/quantization/algorithms/fast_bias_correction/onnx_backend.py code, I am encountering an error that says 'Could not find the bias value of the node'. Do I now have to add a bias of 0 to all convolutional layers that do not have a bias, in order to avoid this error during quantization?

The code for quantization is:

import torch
import numpy as np
import onnx
from torchvision import datasets
from torchvision import transforms

import nncf

model_path = "/home/fp32_mainbody_onnx_bs/const_shape_pp_main_body.onnx"

normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
val_dataset = datasets.ImageFolder(
    root=f"/home/resize_images_400",
    transform=transforms.Compose(
        [
            transforms.Resize(640),
            transforms.ToTensor(),
            normalize,
        ]
    ),
)

val_loader = torch.utils.data.DataLoader(val_dataset, batch_size=1, shuffle=False)
model = onnx.load(model_path)

def transform_fn(data_item):
    images, _ = data_item
    scale = np.array([640 / 1800, 640 / 1800], dtype='float32').reshape(1, 2)
    return {input_img_name: images.numpy(),input_scale_name: scale}
    
calibration_dataset = nncf.Dataset(val_loader, transform_fn)
onnx_quantized_model = nncf.quantize(model, calibration_dataset, subset_size=400)

int8_model_path = f"/home/int8_nncf_quant/int8_main_body.onnx"
onnx.save(onnx_quantized_model, int8_model_path)

The error message is:

Traceback (most recent call last):
File "nncf_quant_mainbody.py", line 40, in
onnx_quantized_model = nncf.quantize(model, calibration_dataset, subset_size=400)
File "/home/.local/lib/python3.8/site-packages/nncf/quantization/quantize_model.py", line 93, in quantize
return quantize_impl(
File "/home/.local/lib/python3.8/site-packages/nncf/telemetry/decorator.py", line 71, in wrapped
retval = fn(*args, **kwargs)
File "/home/.local/lib/python3.8/site-packages/nncf/onnx/quantization/quantize_model.py", line 68, in quantize_impl
quantized_model = quantization_algorithm.apply(model, dataset=calibration_dataset)
File "/home/.local/lib/python3.8/site-packages/nncf/quantization/algorithms/algorithm.py", line 58, in apply
return self._apply(model, statistic_points=None, dataset=dataset)
File "/home/.local/lib/python3.8/site-packages/nncf/quantization/algorithms/post_training/algorithm.py", line 188, in _apply
modified_model = algorithm.apply(modified_model, statistic_points)
File "/home/.local/lib/python3.8/site-packages/nncf/quantization/algorithms/algorithm.py", line 63, in apply
return self._apply(model, statistic_points)
File "/home/.local/lib/python3.8/site-packages/nncf/quantization/algorithms/fast_bias_correction/algorithm.py", line 136, in _apply
for node, bias_value in tqdm(list(node_and_bias_value), desc="Biases correction"):
File "/home/.local/lib/python3.8/site-packages/nncf/quantization/algorithms/fast_bias_correction/algorithm.py", line 128, in
(node, self._backend_entity.get_bias_value(node, nncf_graph, model))
File "/home/.local/lib/python3.8/site-packages/nncf/quantization/algorithms/fast_bias_correction/onnx_backend.py", line 86, in get_bias_value
return get_bias_value(node, model)
File "/home/.local/lib/python3.8/site-packages/nncf/onnx/graph/node_utils.py", line 60, in get_bias_value
raise RuntimeError("Could not find the bias value of the node")
RuntimeError: Could not find the bias value of the node

[l-bat]

@edition3234 could you please attach const_shape_pp_main_body.onnx model to reproduce issue?

[edition3234]

@l-bat I have uploaded the model to Google Drive. Here is the link: https://drive.google.com/file/d/14Xh9Uj4kEjxuWmGewGyT9b9dIHGjUp8V/view?usp=drive_link

[kshpv]

Hello, @edition3234!
Thank you for the described problem.
I could reproduce your issue with the code and model provided. I made a PR #1940 that fixes the problem. When it is merged please use the latest version of the develop branch.

[edition3234]

Hello,@kshpv
I have modified the original model, added bias to the convolutional layer without bias, and ignored some error messages from the quantized convolutional layer. The problem of convolutional layer error messages seems to have been temporarily bypassed. But now the error message is: "Error: Duplicate definition-site for (p2o.out.5)." where 'p2o.out.5' is the output of a concat layer named 'p2o.Concat.4'. 'p2o.Concat.4' is given to two different convolutional layers. Now this error message seems to say that the concat layer cannot have two identical outputs, but it should be feasible. Moreover, I used the onnx.checker.check_model() function to check my modified model, and there was no error message,What could be causing this issue? Is it a problem with NNCF? I'm completely at a loss.

I have uploaded the new model to Google Drive. Here is the link:
https://drive.google.com/file/d/1Mb39MGG4B91kkGEyr7Jd3JuYRqXzi7dO/view?usp=sharing

The error message is:

INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, onnx, openvino
image
scale_factor
WARNING:nncf:ONNX models with 10 < opset version < 13 do not support per-channel quantization. Per-tensor quantization will be applied.
INFO:nncf:8 ignored nodes was found by name in the NNCFGraph
INFO:nncf:34 ignored nodes was found by types in the NNCFGraph
INFO:nncf:Not adding activation input quantizer for operation: 730 p2o.Conv.24
731 p2o.Relu.0

INFO:nncf:Not adding activation input quantizer for operation: 732 p2o.Conv.25
733 p2o.HardSigmoid.0

INFO:nncf:Not adding activation input quantizer for operation: 756 p2o.Conv.28
757 p2o.Relu.1

INFO:nncf:Not adding activation input quantizer for operation: 758 p2o.Conv.29
759 p2o.HardSigmoid.1

INFO:nncf:Not adding activation input quantizer for operation: 804 p2o.Concat.0
INFO:nncf:Not adding activation input quantizer for operation: 855 p2o.Concat.2
INFO:nncf:Not adding activation input quantizer for operation: 869 p2o.Concat.4
INFO:nncf:Not adding activation input quantizer for operation: 920 p2o.Concat.6
INFO:nncf:Not adding activation input quantizer for operation: 1216 p2o.Conv.74
INFO:nncf:Not adding activation input quantizer for operation: 951 p2o.Concat.8
INFO:nncf:Not adding activation input quantizer for operation: 1002 p2o.Concat.10
INFO:nncf:Not adding activation input quantizer for operation: 1033 p2o.Concat.12
INFO:nncf:Not adding activation input quantizer for operation: 1084 p2o.Concat.14
INFO:nncf:Not adding activation input quantizer for operation: 1308 p2o.Conv.83
INFO:nncf:Not adding activation input quantizer for operation: 1400 p2o.Conv.92
INFO:nncf:Not adding activation input quantizer for operation: 1492 p2o.Conv.101
Statistics collection: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:03<00:00, 1.11it/s]
Biases correction: 31%|████████████████████████████████ | 29/94 [01:36<03:36, 3.33s/it]
Traceback (most recent call last):
File "nncf_quant_mainbody.py", line 43, in
onnx_quantized_model = nncf.quantize(model, calibration_dataset, fast_bias_correction=False,subset_size=4,
File "/home/.local/lib/python3.8/site-packages/nncf/quantization/quantize_model.py", line 93, in quantize
return quantize_impl(
File "/home/.local/lib/python3.8/site-packages/nncf/telemetry/decorator.py", line 71, in wrapped
retval = fn(*args, **kwargs)
File "/home/.local/lib/python3.8/site-packages/nncf/onnx/quantization/quantize_model.py", line 68, in quantize_impl
quantized_model = quantization_algorithm.apply(model, dataset=calibration_dataset)
File "/home/.local/lib/python3.8/site-packages/nncf/quantization/algorithms/algorithm.py", line 58, in apply
return self._apply(model, statistic_points=None, dataset=dataset)
File "/home/.local/lib/python3.8/site-packages/nncf/quantization/algorithms/post_training/algorithm.py", line 188, in _apply
modified_model = algorithm.apply(modified_model, statistic_points)
File "/home/.local/lib/python3.8/site-packages/nncf/quantization/algorithms/algorithm.py", line 63, in apply
return self._apply(model, statistic_points)
File "/home/.local/lib/python3.8/site-packages/nncf/quantization/algorithms/bias_correction/algorithm.py", line 160, in _apply
bias_shift = self._compute_bias_shift(node, model_copy_subgraph, feed_dicts, statistic_points)
File "/home/.local/lib/python3.8/site-packages/nncf/quantization/algorithms/bias_correction/algorithm.py", line 335, in _compute_bias_shift
engine = EngineFactory.create(model)
File "/home/.local/lib/python3.8/site-packages/nncf/common/factory.py", line 87, in create
return ONNXEngine(model)
File "/home/.local/lib/python3.8/site-packages/nncf/onnx/engine.py", line 29, in init
self.sess = rt.InferenceSession(serialized_model, **rt_session_options)
File "/home/.local/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 360, in init
self._create_inference_session(providers, provider_options, disabled_optimizers)
File "/home/.local/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 399, in _create_inference_session
sess = C.InferenceSession(session_options, self._model_bytes, False, self._read_config_from_model)
onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Error: Duplicate definition-site for (p2o.out.5).

quant code:

import torch
import numpy as np
import onnx
from torchvision import datasets
from torchvision import transforms

import nncf


model_path = "/home/onnx_modifier/onnx-modifier-master/modified_onnx/modified_modified_model.onnx"


normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
val_dataset = datasets.ImageFolder(
    root=f"/home/1/resize_images_400",
    transform=transforms.Compose(
        [
            transforms.Resize(640),
            transforms.ToTensor(),
            normalize,
        ]
    ),
)

val_loader = torch.utils.data.DataLoader(val_dataset, batch_size=1, shuffle=False)
model = onnx.load(model_path)

onnx.checker.check_model(model)

input_names = [input.name for input in model.graph.input]
input_img_name = input_names[0]
input_scale_name = input_names[1]
print(input_img_name)
print(input_scale_name)

def transform_fn(data_item):
    images, _ = data_item
    scale = np.array([640 / 1800, 640 / 1800], dtype='float32').reshape(1, 2)
    return {input_img_name: images.numpy(),input_scale_name: scale}
    
calibration_dataset = nncf.Dataset(val_loader, transform_fn)
onnx_quantized_model = nncf.quantize(model, calibration_dataset, fast_bias_correction=False,subset_size=4,
        ignored_scope=nncf.IgnoredScope(
            types=["Reshape","Slice","Concat"],  # ignore operations
            names=[
                "p2o.Conv.29",  # in the post-processing subgraph
                "p2o.Conv.28",
                "p2o.Conv.74",
                "p2o.Conv.83",
                "p2o.Conv.92",
                "p2o.Conv.101",
                "p2o.Conv.24",
                "p2o.Conv.25",
            ],
        ),
    )

int8_model_path = f"/home/1/int8_nncf_quant/int8_main_body.onnx"
onnx.save(onnx_quantized_model, int8_model_path)

[edition3234]

Should I add two identity nodes between 'p2o.Concat.4' and the two convolutional layers so that each convolutional layer has a unique input?

[edition3234]

@kshpv
I added two Identity layers between the Concat layer and the two convolution layers connected to its output. Now, there are no more errors, but I don't understand why the size before quantization was 28.8m, and after quantization, it increased to 29m. Could there be a problem with my quantization code?

A model that can work normally：
https://drive.google.com/file/d/1oe54nY8u0ePukBaBK7zGlrGVRwSK2aPh/view?usp=sharing

The model after quantization：
https://drive.google.com/file/d/1SugtTDsQZUkCH7ArAo7tm5j0yLSnXXH2/view?usp=sharing

code：

import torch
import numpy as np
import onnx
from torchvision import datasets
from torchvision import transforms

import nncf

model_path = "/home/onnx_modifier/onnx-modifier-master/modified_onnx/1.onnx"


normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
val_dataset = datasets.ImageFolder(
    root=f"/home/1/resize_images_400",
    transform=transforms.Compose(
        [
            transforms.Resize(640),
            transforms.ToTensor(),
            normalize,
        ]
    ),
)

val_loader = torch.utils.data.DataLoader(val_dataset, batch_size=1, shuffle=False)
model = onnx.load(model_path)

onnx.checker.check_model(model)

input_names = [input.name for input in model.graph.input]
input_img_name = input_names[0]
input_scale_name = input_names[1]
print(input_img_name)
print(input_scale_name)

def transform_fn(data_item):
    images, _ = data_item
    scale = np.array([640 / 1800, 640 / 1800], dtype='float32').reshape(1, 2)
    return {input_img_name: images.numpy(),input_scale_name: scale}
    
calibration_dataset = nncf.Dataset(val_loader, transform_fn)
onnx_quantized_model = nncf.quantize(model, calibration_dataset, fast_bias_correction=False,subset_size=400,
        ignored_scope=nncf.IgnoredScope(
            types=["Reshape","Slice","Concat"],  # ignore operations
            names=[
                "p2o.Conv.29",  # in the post-processing subgraph
                "p2o.Conv.28",
                "p2o.Conv.74",
                "p2o.Conv.83",
                "p2o.Conv.92",
                "p2o.Conv.101",
                "p2o.Conv.24",
                "p2o.Conv.25",
            ],
        ),
    )

int8_model_path = f"/home/1/int8_nncf_quant/int8_main_body.onnx"
onnx.save(onnx_quantized_model, int8_model_path)

[kshpv]

@edition3234 could you try quantization on the branch? - #1940
It also introduces some fixes some issues with the quantization scheme. Your model after quantization does not have the optimal quantization scheme.

This branch works fine for me with the original model that you provided. https://drive.google.com/file/d/14Xh9Uj4kEjxuWmGewGyT9b9dIHGjUp8V/view?usp=drive_link

I don't understand why the size before quantization was 28.8m, and after quantization, it increased to 29m.

This is correct. We don't save weights at int8 precision, while the model is inferred at int8 precision.
Therefore model size is not reduced.

[edition3234]

Thank you for your help!!!
With the version you released, the original model can also be quantized normally, but the accuracy has dropped a lot, to the point where it can't be used. I am considering using 'Quantizing with accuracy control' or other methods.

[kshpv]

Thank you for your help!!! With the version you released, the original model can also be quantized normally, but the accuracy has dropped a lot, to the point where it can't be used. I am considering using 'Quantizing with accuracy control' or other methods.

Unfortunately, quantize_with_accuracy_control is not supported for ONNX backend.
What you can do instead:

1. Increase subset_size
2. Ignore some layers for quantization manually. I suggest starting to ignore layers from the end of the network. Because, often, quantization of the last layers can drop accuracy.

[edition3234]

Can I convert the ONNX model to an IR model and then use quantize_with_accuracy_control?

If it's possible, I can modify the part that loads the ONNX model to use OpenVINO to load the IR model, which shouldn't be too difficult. Currently, I'm also trying other methods, but the quantization results have been consistently poor, possibly due to issues with the model structure.

[kshpv]

Can I convert the ONNX model to an IR model and then use quantize_with_accuracy_control?

Yes, you can. In this way, you can reuse IgnoredScope from IR and map it to ONNX.

If it's possible, I can modify the part that loads the ONNX model to use OpenVINO to load the IR model, which shouldn't be too difficult. Currently, I'm also trying other methods, but the quantization results have been consistently poor, possibly due to issues with the model structure.

What issues with model structure do you mean?

Note: You also can quantize the IR model and infer it through OpenVINO

[edition3234]

What issues with model structure do you mean?

I have just started exploring quantization, and I suspect that certain structures may lead to poor quantization results. This is because I have seen some relatively simple structures that can be easily quantized into int8 with decent performance.

I will try to ignore the post-quantization processing and increase the amount of quantized data. Once again, thank you for your help.