Issue on ipex.optimize: `NotImplementedError: argument of type`

[gera-aldama]

Describe the bug

I have the following classes:

class BidirectionalLSTM(nn.Module):

    def __init__(self, nIn, nHidden, nOut):
        super(BidirectionalLSTM, self).__init__()

        self.rnn = nn.LSTM(nIn, nHidden, bidirectional=True)
        self.embedding = nn.Linear(nHidden * 2, nOut)

    def forward(self, input):
        recurrent, _ = self.rnn(input)
        T, b, h = recurrent.size()
        t_rec = recurrent.view(T * b, h)

        output = self.embedding(t_rec)  # [T * b, nOut]
        output = output.view(T, b, -1)
        return output


class CRNN(nn.Module):

    def __init__(self, imgH, nc, nclass, nh, leakyRelu=False):
        super(CRNN, self).__init__()
        assert imgH % 16 == 0, 'imgH has to be a multiple of 16'  # nosec

        # 1x32x128
        self.conv1 = nn.Conv2d(nc, 64, 3, 1, 1)
        self.relu1 = nn.ReLU(True)
        self.pool1 = nn.MaxPool2d(2, 2)

        # 64x16x64
        self.conv2 = nn.Conv2d(64, 128, 3, 1, 1)
        self.relu2 = nn.ReLU(True)
        self.pool2 = nn.MaxPool2d(2, 2)

        # 128x8x32
        self.conv3_1 = nn.Conv2d(128, 256, 3, 1, 1)
        self.bn3 = nn.BatchNorm2d(256)
        self.relu3_1 = nn.ReLU(True)
        self.conv3_2 = nn.Conv2d(256, 256, 3, 1, 1)
        self.relu3_2 = nn.ReLU(True)
        self.pool3 = nn.MaxPool2d((2, 2), (2, 1), (0, 1))

        # 256x4x16
        self.conv4_1 = nn.Conv2d(256, 512, 3, 1, 1)
        self.bn4 = nn.BatchNorm2d(512)
        self.relu4_1 = nn.ReLU(True)
        self.conv4_2 = nn.Conv2d(512, 512, 3, 1, 1)
        self.relu4_2 = nn.ReLU(True)
        self.pool4 = nn.MaxPool2d((2, 2), (2, 1), (0, 1))

        # 512x2x16
        self.conv5 = nn.Conv2d(512, 512, 2, 1, 0)
        self.bn5 = nn.BatchNorm2d(512)
        self.relu5 = nn.ReLU(True)

        # 512x1x16

        self.rnn = nn.Sequential(
            BidirectionalLSTM(512, nh, nh),
            BidirectionalLSTM(nh, nh, nclass))

    def forward(self, input):
        # conv features
        x = self.pool1(self.relu1(self.conv1(input)))
        x = self.pool2(self.relu2(self.conv2(x)))
        x = self.pool3(self.relu3_2(self.conv3_2(self.relu3_1(self.bn3(self.conv3_1(x))))))
        x = self.pool4(self.relu4_2(self.conv4_2(self.relu4_1(self.bn4(self.conv4_1(x))))))
        conv = self.relu5(self.bn5(self.conv5(x)))
        # print(conv.size())

        b, c, h, w = conv.size()
        assert h == 1, "the height of conv must be 1"  # nosec
        conv = conv.squeeze(2)
        conv = conv.permute(2, 0, 1)  # [w, b, c]

        # rnn features
        output = self.rnn(conv)

        return output

And when trying to run the following code (also using Intel Neural Compressor):

fp32_crnn_model = crnn.CRNN(config.imgH, config.nc, config.nclass, config.nh)
fp32_crnn_model.load_state_dict(torch.load(fp32_crnn_model_path, map_location="cpu"))
quantized_crnn_model = load(quantized_crnn_model_path, fp32_crnn_model)

fp32_crnn_model.eval()
quantized_crnn_model.eval()

import intel_extension_for_pytorch as ipex
quantized_crnn_model = ipex.optimize(quantized_crnn_model)

I get on ipex.optimize:
NotImplementedError: argument of type: <class 'crnn.CRNN'>

Versions

/root/miniconda/envs/invoice/lib/python3.9/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: '/root/miniconda/envs/invoice/lib/python3.9/site-packages/torchvision/image.so: undefined symbol: _ZN5torch3jit17parseSchemaOrNameERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
  warn(
Collecting environment information...
PyTorch version: 2.0.1+cpu
PyTorch CXX11 ABI: No
IPEX version: 2.0.100+cpu
IPEX commit: 6a341a3
Build type: Release

OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: N/A
IGC version: N/A
CMake version: N/A
Libc version: glibc-2.35

Python version: 3.9.18 (tags/v3.9.18-26-g6b320c3b2f6-dirty:6b320c3b2f6, Sep 28 2023, 00:35:27)  [GCC 13.2.0] (64-bit runtime)
Python platform: Linux-4.15.0-191-generic-x86_64-with-glibc2.35
Is XPU available: False
DPCPP runtime version: N/A
MKL version: N/A
GPU models and configuration: 

Intel OpenCL ICD version: N/A
Level Zero version: N/A

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Address sizes:                   46 bits physical, 48 bits virtual
Byte Order:                      Little Endian
CPU(s):                          112
On-line CPU(s) list:             0-111
Vendor ID:                       GenuineIntel
Model name:                      Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz
CPU family:                      6
Model:                           85
Thread(s) per core:              2
Core(s) per socket:              28
Socket(s):                       2
Stepping:                        4
CPU max MHz:                     3800.0000
CPU min MHz:                     1000.0000
BogoMIPS:                        5000.00
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear flush_l1d arch_capabilities
Virtualization:                  VT-x
L1d cache:                       1.8 MiB (56 instances)
L1i cache:                       1.8 MiB (56 instances)
L2 cache:                        56 MiB (56 instances)
L3 cache:                        77 MiB (2 instances)
NUMA node(s):                    2
NUMA node0 CPU(s):               0-27,56-83
NUMA node1 CPU(s):               28-55,84-111
Vulnerability Itlb multihit:     KVM: Mitigation: Split huge pages
Vulnerability L1tf:              Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds:               Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown:          Mitigation; PTI
Vulnerability Mmio stale data:   Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Mitigation; Clear CPU buffers; SMT vulnerable

Versions of relevant libraries:
[pip3] intel-extension-for-pytorch==2.0.100+cpu
[pip3] numpy==1.24.3
[pip3] torch==2.0.1+cpu
[pip3] torchvision==0.15.2a0+cxx11.abi
[conda] intel-extension-for-pytorch 2.0.100              py39_cpu_0    intel
[conda] mkl                       2024.0.0            intel_49656    intel
[conda] mkl-dpcpp                 2024.0.0            intel_49656    intel
[conda] mkl-include               2024.0.0            intel_49656    intel
[conda] mkl-service               2.4.0           py39h3539a15_40    intel
[conda] mkl_fft                   1.3.6           py39h1d81ff8_61    intel
[conda] mkl_random                1.2.2           py39h5a378b4_81    intel
[conda] mkl_umath                 0.1.1           py39h2b1685c_91    intel
[conda] numpy                     1.24.3           py39ha320b8e_5    intel
[conda] numpy-base                1.24.3           py39hbac2b65_5    intel
[conda] onemkl-sycl-blas          2024.0.0            intel_49656    intel
[conda] onemkl-sycl-datafitting   2024.0.0            intel_49656    intel
[conda] onemkl-sycl-dft           2024.0.0            intel_49656    intel
[conda] onemkl-sycl-lapack        2024.0.0            intel_49656    intel
[conda] onemkl-sycl-rng           2024.0.0            intel_49656    intel
[conda] onemkl-sycl-sparse        2024.0.0            intel_49656    intel
[conda] onemkl-sycl-stats         2024.0.0            intel_49656    intel
[conda] onemkl-sycl-vm            2024.0.0            intel_49656    intel
[conda] pytorch                   2.0.1                py39_cpu_0    intel
[conda] pytorch-mutex             1.0                         cpu    intel
[conda] torchvision               0.15.2                   py39_0    intel

[kta-intel]

Thanks @gera-aldama. We'll take a look at this. Can you also provide the script or code-snippet that you used to quantize the model?

Alternatively, ipex also has built in quantization api that I recommend

[gera-aldama]

Sure.

Quantization:

from neural_compressor.experimental import Quantization, common 

config_path =config.yml_quantized
    
crnn = crnn.CRNN(config.imgH, config.nc, config.nclass, config.nh).to("cpu")
crnn.load_state_dict(torch.load(model_path, map_location="cpu"))
dataset = Dataset()
quantizer = Quantization(config_path)
quantizer.model = common.Model(crnn)
quantizer.calib_dataloader = common.DataLoader(dataset)
q_model = quantizer.fit()
q_model.save(FLAGS.output_path)

With yml_quantized being:

# conf.yaml
version: 1.0

model:
    name: CRNN
    framework: pytorch_fx
evaluation:
  accuracy:
    metric:
      topk: 1
  performance:
    configs:
      cores_per_instance: 4
      num_of_instance: 1
quantization:
  calibration:
    sampling_size: 200

tuning:
  accuracy_criterion:
    relative:  0.01

[kta-intel]

Hi @gera-aldama, can you set the framework in the config file to use the ipex pytorch backend? i.e framework: pytorch_ipex, rerun quantization, and try again with your original inference script?

[gera-aldama]

Hello, configuring framework: pytorch_ipex solved my issue. Thanks a lot! @kta-intel Will close issue.