performance and error issue with INT8 quantization example

[chtlp]

Hi, I am trying the the new quantization example in the master branch of MxNet, however the original code seems to have much slower inference speed, while the revised code throws an error with cuDNN.
https://github.com/apache/incubator-mxnet/blob/master/example/quantization/

Description

The example works fine when saving when saving the matrices in the cpu() context:
https://github.com/apache/incubator-mxnet/blob/master/example/quantization/imagenet_gen_qsym.py#L49

However, the inference speed actually got slower on P100/GTX 1080ti:

Original model

CUDA_VISIBLE_DEVICES=0 python imagenet_inference.py --symbol-file=./model/imagenet1k-resnet-152-symbol.json --param-file=./model/imagenet1k-resnet-152-0000.params --rgb-mean=0,0,0 --num-skipped-batches=50 --num-inference-batches=500 --dataset=./data/val_256_q90.rec

INFO:logger:Finished inference with 16000 images
INFO:logger:Finished with 252.172165 images per second
INFO:logger:('accuracy', 0.772375)
INFO:logger:('top_k_accuracy_5', 0.93)

Quantized model

CUDA_VISIBLE_DEVICES=0 python imagenet_inference.py --symbol-file=./model/imagenet1k-resnet-152-quantized-5batches-naive-symbol.json --param-file=./model/imagenet1k-resnet-152-quantized-0000.params --rgb-mean=0,0,0 --num-skipped-batches=50 --num-inference-batches=500 --dataset=./data/val_256_q90.rec

INFO:logger:Finished inference with 16000 images
INFO:logger:Finished with 105.886662 images per second
INFO:logger:('accuracy', 0.7596875)
INFO:logger:('top_k_accuracy_5', 0.9234375)

When I tried to changed the saved context to gpu() on https://github.com/apache/incubator-mxnet/blob/master/example/quantization/imagenet_gen_qsym.py#L49
cuDNN threw an exception during the inference:

 File "/opt/home/linpengt/anaconda3/lib/python3.6/site-packages/mxnet/base.py", line 149, in check_call
raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: [15:40:54] src/operator/quantization/quantized_conv.cu:242: Check failed: e == CUDNN_STATUS_SUCCESS (6 vs. 0) cuDNN: CUDNN_STATUS_ARCH_MISMATCH

Just wondering if the speed decrease and the error is as expected, or if there is a way to fix that. Thanks!
​

Environment info (Required)

----------Python Info----------
Version      : 3.6.3
Compiler     : GCC 7.2.0
Build        : ('default', 'Oct 13 2017 12:02:49')
Arch         : ('64bit', '')
------------Pip Info-----------
Version      : 10.0.1
Directory    : /opt/home/linpengt/anaconda3/lib/python3.6/site-packages/pip
----------MXNet Info-----------
Version      : 1.2.0
Directory    : /opt/home/linpengt/anaconda3/lib/python3.6/site-packages/mxnet
Commit Hash   : f02caae5a06c84d2e764f3a4c4283cf838e3d69f
----------System Info----------
Platform     : Linux-4.4.0-121-generic-x86_64-with-debian-stretch-sid
system       : Linux
node         : dev0006.internal.moqi.ai
release      : 4.4.0-121-generic
version      : #145-Ubuntu SMP Fri Apr 13 13:47:23 UTC 2018
----------Hardware Info----------
machine      : x86_64
processor    : x86_64
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                56
On-line CPU(s) list:   0-55
Thread(s) per core:    2
Core(s) per socket:    14
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 79
Model name:            Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz
Stepping:              1
CPU MHz:               1256.125
CPU max MHz:           3500.0000
CPU min MHz:           1200.0000
BogoMIPS:              5201.68
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              35840K
NUMA node0 CPU(s):     0-13,28-41
NUMA node1 CPU(s):     14-27,42-55
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb invpcid_single intel_pt retpoline kaiser tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdseed adx smap xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts
----------Network Test----------
Setting timeout: 10
Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0240 sec, LOAD: 5.9882 sec.
Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.0260 sec, LOAD: 7.4869 sec.
Timing for Gluon Tutorial(cn): https://zh.gluon.ai, DNS: 0.0157 sec, LOAD: 0.8470 sec.
Timing for FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.4073 sec, LOAD: 2.3378 sec.
Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0164 sec, LOAD: 7.5267 sec.
Timing for Conda: https://repo.continuum.io/pkgs/free/, DNS: 0.0268 sec, LOAD: 0.9382 sec.

GPU: P100/1080ti
MxNet: mxnet-cu90 1.2.0b20180509
CUDA 9.0, cuDNN 7.1

Error Message:

[11:33:01] src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:107: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
Traceback (most recent call last):
  File "imagenet_inference.py", line 176, in <module>
    max_num_examples=num_inference_images, logger=logger)
  File "imagenet_inference.py", line 86, in score
    mod.update_metric(m, batch.label)
  File "/opt/home/linpengt/anaconda3/lib/python3.6/site-packages/mxnet/module/module.py", line 757, in update_metric
    self._exec_group.update_metric(eval_metric, labels)
  File "/opt/home/linpengt/anaconda3/lib/python3.6/site-packages/mxnet/module/executor_group.py", line 616, in update_metric
    eval_metric.update_dict(labels_, preds)
  File "/opt/home/linpengt/anaconda3/lib/python3.6/site-packages/mxnet/metric.py", line 132, in update_dict
    self.update(label, pred)
  File "/opt/home/linpengt/anaconda3/lib/python3.6/site-packages/mxnet/metric.py", line 418, in update
    pred_label = pred_label.asnumpy().astype('int32')
  File "/opt/home/linpengt/anaconda3/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line 1890, in asnumpy
    ctypes.c_size_t(data.size)))
  File "/opt/home/linpengt/anaconda3/lib/python3.6/site-packages/mxnet/base.py", line 149, in check_call
    raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: [11:33:01] src/operator/quantization/quantized_conv.cu:242: Check failed: e == CUDNN_STATUS_SUCCESS (6 vs. 0) cuDNN: CUDNN_STATUS_ARCH_MISMATCH

Stack trace returned 10 entries:
[bt] (0) /opt/home/linpengt/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x32dbea) [0x7f2443ed1bea]
[bt] (1) /opt/home/linpengt/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x32e211) [0x7f2443ed2211]
[bt] (2) /opt/home/linpengt/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2c73251) [0x7f2446817251]
[bt] (3) /opt/home/linpengt/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2c92146) [0x7f2446836146]
[bt] (4) /opt/home/linpengt/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x247a110) [0x7f244601e110]
[bt] (5) /opt/home/linpengt/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x248292c) [0x7f244602692c]
[bt] (6) /opt/home/linpengt/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2462ca4) [0x7f2446006ca4]
[bt] (7) /opt/home/linpengt/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2466b43) [0x7f244600ab43]
[bt] (8) /opt/home/linpengt/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2466d96) [0x7f244600ad96]
[bt] (9) /opt/home/linpengt/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x24633b4) [0x7f24460073b4]

Minimum reproducible example

Run quantization as in https://github.com/apache/incubator-mxnet/blob/master/example/quantization/launch_quantize.sh#L24

python imagenet_gen_qsym.py --model=imagenet1k-resnet-152 --calib-dataset=./data/val_256_q90.rec --num-calib-batches=5 --calib-mode=naive

Then run inference as in https://github.com/apache/incubator-mxnet/blob/master/example/quantization/launch_inference.sh#L26

python imagenet_gen_qsym.py --model=imagenet1k-resnet-152 --calib-dataset=./data/val_256_q90.rec --num-calib-batches=5 --calib-mode=naive

[vdantu]

@sandeep-krishnamurthy : Could you please tag this as "Flaky", "Python", "Example", "Question".

[reminisce]

The purpose of the quantization implementation of this version is keeping accuracy under control, i.e. minimizing the accuracy loss when converting FP32 models to INT8. We are seeking other paths for improving the speed.

It's not right to change the context in that function. It's used to serialize params to disk, but doesn't mean the params does not exist on GPU when you load the model. It's all handled well by the Module interface.

[chtlp]

@reminisce OK, so the performance decrease on GPU is expected? Then it's better to add some comments to the example, otherwise it would be very confusing.

[reminisce]

@chtlp Yes, the slow down is expected. We would make the doc clearer. Thanks.

[ThomasDelteil]

@reminisce have the docs been updated to let users know about the slowness and maybe explain the source of it?
@reminisce can you remove the Example and Question tags now that the question has been answered
Thanks!