Multithreading scalability on Ernie INT8 with oneDNN and Resnet50 without MKLDNN on CPU

[lidanqing-vv]

paddle-deepmd multithreading without MKLDNN is worse than other frameworks

• Deepmd multithreading issue could been exported to a simple demo + Paddle without lammps, which is in deep_md_test.zip. Below text is just comparing tensorflow and Paddle, to reproduce deepmd + Paddle multithreading issue, you can skip below texts and go directly to Multithreading scalability on Ernie INT8 with oneDNN and Resnet50 without MKLDNN on CPU #43215 (comment)

• Paddle Deepmd official website:
  https://github.com/X4Science/paddle-deepmd

• To easier reproduce multhreading issue (that paddle-deepmd multithreading is worse than tf deepmd-kit)
  https://github.com/lidanqing-intel/deepmd-kit/blob/paddle-test/README.md

• Reproduction environments:
  Paddle version: eca6638
  Test machine: Intel(R) Xeon(R) Platinum 8352Y CPU @ 2.20GHz (ICX)
  Performance result:
  

• Reproduce paddle-deepmd multithreading

git clone https://github.com/lidanqing-intel/deepmd-kit.git
git checkout paddle-test
bash compile_paddle.sh
source .bashrc
bash compile_deepmd.sh
bash compile_lammps.sh
cd setting/lmp
# single thread, single mpi and multi threads, multi mpi
bash lmp_pp.sh

• Reproduce tf-test multithreading

git clone https://github.com/lidanqing-intel/deepmd-kit.git
git checkout tf-test
bash compile_tf.sh
source .bashrc
bash compile_deepmd.sh
bash compile_lammps.sh
cd setting/lmp_tf
bash lmp_tf.sh

PaddlePaddle- ernie3.0 INT8 with MKLDNN, try to improve multithreading scalability

Paddle: 0d719718b308587efcb6b3547f925582a8009176
model download https://paddlenlp.bj.bcebos.com/models/transformers/ernie_3.0/ernie3.0_medium_inference_models.zip
. After model distraction, there will be 4 files， (float32.pdmodel, float32.pdiparams) and (int8.pdmodel, int8.pdiparams)
, which is float32 model and int8 quant model.

git clone https://github.com/PaddlePaddle/PaddleNLP.git
cd PaddleNLP
pip install -r requirements.txt
python setup.py install
cd model_zoo/ernie-3.0

• Ernie-3.0 FP32 mkldnn, 1 thread on ICX is 65.45 QPS
  python infer.py --task_name tnews --model_path /home/guest/PaddleNLP/model_zoo/ernie-3.0/ernie-3.0/float32--perf --device cpu --num_threads 1

• Ernie-3.0 INT8 mkldnn, 1 thread on ICX is 153.77 QPS
  python infer.py --task_name tnews --model_path /home/guest/PaddleNLP/model_zoo/ernie-3.0/ernie-3.0/int8 --perf --device cpu --num_threads 1 --enable_quantize

[paddle-bot-old]

您好，我们已经收到了您的问题，会安排技术人员尽快解答您的问题，请耐心等待。请您再次检查是否提供了清晰的问题描述、复现代码、环境&版本、报错信息等。同时，您也可以通过查看官网API文档、常见问题、历史Issue、AI社区来寻求解答。祝您生活愉快～

Hi! We've received your issue and please be patient to get responded. We will arrange technicians to answer your questions as soon as possible. Please make sure that you have posted enough message to demo your request. You may also check out the API，FAQ，Github Issue and AI community to get the answer.Have a nice day!

[jczaja]

@yaomichael , @lidanqing-intel

Just to let you know that we are working on this one. So to effectively utilize parallelization there need to be more data to parallelize for example batches of data (batch size > 1) . And we found that some implementations of oneDNN for bigger batches do not work as fast as they should be . So we are looking at this now. After we narrow down problems with big batches execution we will work on actual poor multi-threading scalability.

[jczaja]

@jiangjiajun , @lidanqing-intel , @yaomichael
So we started working on fixing poor performance of execution when batch_size is bigger than 1. We narrowed down couple of problems and managed to fix some of them. To be more precise mostly FC (inner product) oneDNN operator's kernel was performing poorly and then convoluution for int8 was also under performing. First fix to FC (more to follow) is here: #44078 . Fix to convolution was already fixed in oneDNN so we are waiting till stable version is released (oneDNN 2.7) to introduce it to paddlepaddle.

Now we start to look at actual multi-threading issues for supported models .

[jczaja]

@jiangjiajun , @lidanqing-intel , @yaomichael
So we merged fixes for problem of poor performance of bigger batches. So I spend some time making measurements for Resnet50 to retest problem. Here is visual representation:

This figure presents a scalability of performance as we use more threads for various batch sizes. For batch size of 100 we can see that for one thread we have a bit over 50 FPS and for eight threads we got result of a bit 400 FPS. Which is almost 8x (almost linear scalability). For batch size : 1 , scalability is poorer. So overall it is not bad e.g. I do not see any big problem for Resnet50 int8 for the processor I used (CLX). As a next step I will either test other (non-convolutional) models like ERNIE int8.