RFC: Best Practices re OPENMP - for training, evaluation and recognition

[Shreeshrii]

For Tesseract 5 what are the best practices regarding OPENMP.

Is it still true:

1. OPENMP is needed for training so build tesseract and training tools with --enable-openmp.
2. For lstmeval (built with --enable-openmp), use OMP_THREAD_LIMIT=1.
3. For recognition with tesseract (built with --enable-openmp), use OMP_THREAD_LIMIT=1.

[stweil]

OPENMP is not needed for training. It even makes things worse for me. Timing results for lstm_squashed_test on AMD EPYC 7502 show that no OPENMP (--disable-openmp) is best, followed by disabled OPENMP (OMP_THREAD_LIMIT=1). Enabled OPENMP is the last and burns a lot of CPU performance for nothing:

# --disable-openmp
real 28.41
user 28.33
sys 0.08
# --enable-openmp
real 33.16
user 129.41
sys 1.46
# --enable-openmp, OMP_THREAD_LIMIT=1
real 32.89
user 32.61
sys 0.28

[amitdo]

The plan is to disable it by default in 5.1.0.

[stweil]

... in autoconf builds. cmake already disables it by default.

[stweil]

Note that even without OPENMP training uses up to two CPU threads, one for training which runs until training is finished and one for evaluation which runs from time to time during the training process.

[amitdo]

The reason for disabling OpenMP is that Tesseract currently uses it inefficiently.

For text recognition the speed benefit for using OpenMP with fast / tessdata (best->int) traineddata is too small while it consumes too much CPU resources.

For training the OpenMP code is even more problematic than the code used for text recognition. I'm not sure how much speed will be lost here.

[Shreeshrii]

Thank you!

no OPENMP is best, followed by disabled OPENMP

Does no OPENMP mean building with --disable-openmp as part of autotools configure?

[stweil]

Yes, currently it is necessary to use configure --disable-openmp. As Amit has written above that should be the default, but I still have no simple code to achieve that.

I updated my comment to be clearer.

[amitdo]

--disable-openmp disables OpenMP at compile time, while OMP_THREAD_LIMIT=1 disables it at runtime. The first method is more efficient, while the second method is more flexible.

[amitdo]

Stefan, for 5.1.0, do you want to keep a way to enable OpenMP with --enable-openmp?

[Shreeshrii]

OPENMP is not needed for training. It even makes things worse for me. Timing results for lstm_squashed_test on AMD EPYC 7502 show that no OPENMP (--disable-openmp) is best, followed by disabled OPENMP (OMP_THREAD_LIMIT=1). Enabled OPENMP is the last and burns a lot of CPU performance for nothing:

Ok. I will try to test training from font scenarios in my tess5train-fonts repo to see if they get similar results.

[Shreeshrii]

lstmeval

Which time figures (real, user, sys) are important? Which scenario is preferable?

no OPENMP (--disable-openmp)

tesseract 5.0.1-19-g44ddde
 leptonica-1.78.0
  libgif 5.1.4 : libjpeg 8d (libjpeg-turbo 1.5.2) : libpng 1.6.34 : libtiff 4.0.9 : zlib 1.2.11 : libwebp 0.6.1 : libopenjp2 2.3.0
 Found NEON
 Found libarchive 3.2.2 zlib/1.2.11 liblzma/5.2.2 bz2lib/1.0.6 liblz4/1.7.1
 Found libcurl/7.58.0 NSS/3.35 zlib/1.2.11 libidn2/2.0.4 libpsl/0.19.1 (+libidn2/2.0.4) nghttp2/1.30.0 librtmp/2.3

time -p lstmeval  \
	--verbosity=0 \
	--model data/engFineTuned/tessdata_fast/engFineTuned_0.631000_121_600.traineddata \
	--eval_listfile data/engFineTuned/list.eval 2>&1 | grep "^BCER eval" > data/engFineTuned/tessdata_fast/engFineTuned_0.631000_121_600.eval.log
real 805.37
user 805.34
sys 0.03
time -p lstmeval  \
	--verbosity=0 \
	--model data/engFineTuned/tessdata_fast/engFineTuned_0.028000_156_2000.traineddata \
	--eval_listfile data/engFineTuned/list.eval 2>&1 | grep "^BCER eval" > data/engFineTuned/tessdata_fast/engFineTuned_0.028000_156_2000.eval.log
real 806.56
user 806.49
sys 0.07
time -p lstmeval  \
	--verbosity=0 \
	--model data/engFineTuned/tessdata_fast/engFineTuned_0.558000_125_700.traineddata \
	--eval_listfile data/engFineTuned/list.eval 2>&1 | grep "^BCER eval" > data/engFineTuned/tessdata_fast/engFineTuned_0.558000_125_700.eval.log
real 806.10
user 806.04
sys 0.07

Enabled OPENMP

older version of tesseract 5.0.1 built with --enable-openmp

time -p lstmeval  \
	--verbosity=0 \
	--model data/engFineTuned/tessdata_fast/engFineTuned_0.645000_119_600.traineddata \
	--eval_listfile data/engFineTuned/list.eval 2>&1 | grep "^BCER eval" > data/engFineTuned/tessdata_fast/engFineTuned_0.645000_119_600.eval.log
real 331.53
user 1041.90
sys 9.02
time -p lstmeval  \
	--verbosity=0 \
	--model data/engFineTuned/tessdata_fast/engFineTuned_0.119000_156_1500.traineddata \
	--eval_listfile data/engFineTuned/list.eval 2>&1 | grep "^BCER eval" > data/engFineTuned/tessdata_fast/engFineTuned_0.119000_156_1500.eval.log
real 331.30
user 1042.38
sys 8.55
time -p lstmeval  \
	--verbosity=0 \
	--model data/engFineTuned/tessdata_fast/engFineTuned_0.014000_165_2500.traineddata \
	--eval_listfile data/engFineTuned/list.eval 2>&1 | grep "^BCER eval" > data/engFineTuned/tessdata_fast/engFineTuned_0.014000_165_2500.eval.log
real 331.70
user 1042.77
sys 8.97

[Shreeshrii]

lstmeval - engImpact

No OPENMP

time -p lstmeval  \
	--verbosity=0 \
	--model data/engImpact/tessdata_fast/engImpact_0.489000_152_900.traineddata \
	--eval_listfile data/engImpact/list.eval 2>&1 | grep "^BCER eval" > data/engImpact/tessdata_fast/engImpact_0.489000_152_900.eval.log
real 19.85
user 19.82
sys 0.04

OPENMP

time -p lstmeval  \
	--verbosity=0 \
	--model data/engImpact/tessdata_fast/engImpact_0.489000_152_900.traineddata \
	--eval_listfile data/engImpact/list.eval 2>&1 | grep "^BCER eval" > data/engImpact/tessdata_fast/engImpact_0.489000_152_900.eval.log
real 8.25
user 25.87
sys 0.27