[build] Fix configure breakage from #3194 MKL switch #3216

kkm000 · 2019-04-09T16:13:00Z

Extend the validation of mutual consistency between mathlib-root
switches, and establish MATHLIB as the single ground-truth value
of the chosen matrix algebra library for the remainder of the script.

Fixes #3217.

Establish MATHLIB as the single ground-truth source for the chosen matrix algebra library.

kkm000 · 2019-04-09T16:16:12Z

Quick tests with MKL and ATLAS

(OPENBLAS will be run on Travis, and CLAPACK is too generic):

--mathlib=ATLAS, libatlas-dev not installed

$ ./configure  --mathlib=ATLAS
Configuring KALDI to use ATLAS.
Backing up kaldi.mk to kaldi.mk.bak ...
Checking compiler g++ ...
Checking OpenFst library in /home/kkm/work/kaldi2/tools/openfst ...
Checking cub library in /home/kkm/work/kaldi2/tools/cub ...
Doing OS specific configurations ...
On Linux: Checking for linear algebra header files ...
Using ATLAS as the linear algebra library.
Could not find libatlas.a in any of the generic-Linux places, but we'll try other stuff...
** Failed to configure ATLAS libraries ***
**  ERROR   **
** Configure cannot proceed automatically.
**  If you know that you have ATLAS installed somewhere on your machine, you
** may be able to proceed by replacing [somewhere] in kaldi.mk with a directory.
**  If you have sudo (root) access you could install the ATLAS package on your
** machine, e.g. 'sudo apt-get install libatlas-dev libatlas-base-dev' or
** 'sudo yum install atlas.x86_64' or 'sudo zypper install libatlas3-devel',
** or on cygwin, install atlas from the installer GUI; and then run ./configure
** again.
**
**  Otherwise (or if you prefer OpenBLAS for speed), you could go the OpenBLAS
** route: cd to ../tools, type 'extras/install_openblas.sh', cd back to here,
** and type './configure  --openblas-root=../tools/OpenBLAS/install'

Inastall libatlas-dev, use --mathlib=ATLAS

$ sudo apt install -y libatlas-base-dev
. . .
$ ./configure  --mathlib=ATLAS
Configuring KALDI to use ATLAS.
Backing up kaldi.mk to kaldi.mk.bak ...
Checking compiler g++ ...
Checking OpenFst library in /home/kkm/work/kaldi2/tools/openfst ...
Checking cub library in /home/kkm/work/kaldi2/tools/cub ...
Doing OS specific configurations ...
On Linux: Checking for linear algebra header files ...
Using ATLAS as the linear algebra library.
Successfully configured ATLAS with ATLASLIBS=/usr/lib/x86_64-linux-gnu//libatlas.so.3 /usr/lib/x86_64-linux-gnu//libf77blas.so.3 /usr/lib/x86_64-linux-gnu//libcblas.so.3 /usr/lib/x86_64-linux-gnu//liblapack_atlas.so.3
WARNING: CUDA will not be used! If you have already installed cuda drivers
         and CUDA toolkit, try using --cudatk-dir=... option. CUDA is necessary
         to perform neural net experiment in a realistic time.
Info: configuring Kaldi not to link with Speex (don't worry, it's only needed if you
intend to use 'compress-uncompress-speex', which is very unlikely)
SUCCESS
To compile: make clean -j; make depend -j; make -j
 ... or e.g. -j 10, instead of -j, to use a specified number of CPUs
$ make clean && make depend -j && make -j16 -Otarget
. . .
Done
$ sudo apt remove -y libatlas-base-dev --auto-remove
. . .

No args, MKL present, ATLAS removed.

$ ./configure
Configuring KALDI to use MKL.
Checking compiler g++ ...
Checking OpenFst library in /home/kkm/work/kaldi2/tools/openfst ...
Checking cub library in /home/kkm/work/kaldi2/tools/cub ...
Doing OS specific configurations ...
On Linux: Checking for linear algebra header files ...
Configuring MKL library directory: Found: /opt/intel/mkl/lib/intel64
MKL configured with threading: sequential, libs: -L/opt/intel/mkl/lib/intel64 -Wl,-rpath=/opt/intel/mkl/lib/intel64 -lmkl_intel_lp64  -lmkl_core  -lmkl_sequential
MKL include directory configured as: /opt/intel/mkl/include
Configuring MKL threading as sequential
MKL threading libraries configured as   -ldl -lpthread -lm
Using Intel MKL as the linear algebra library.
Intel(R) Math Kernel Library Version 2019.0.2 Product Build 20190118 for Intel(R) 64 architecture applications
Successfully configured for Linux with MKL libs from /opt/intel/mkl
WARNING: CUDA will not be used! If you have already installed cuda drivers
         and CUDA toolkit, try using --cudatk-dir=... option. CUDA is necessary
         to perform neural net experiment in a realistic time.
Info: configuring Kaldi not to link with Speex (don't worry, it's only needed if you
intend to use 'compress-uncompress-speex', which is very unlikely)
SUCCESS
To compile: make clean -j; make depend -j; make -j
 ... or e.g. -j 10, instead of -j, to use a specified number of CPUs
$ make clean && make depend -j && make -j16 -Otarget
. . .
Done

kkm000 · 2019-04-09T16:45:42Z

@jtrmal, @luitjens, would appreciate if you could review!

luitjens · 2019-04-09T16:58:47Z

src/configure

-unset CLAPACKROOT
-unset OPENBLASROOT
-unset MKLLIBDIR
+#TODO(kkm): Maybe allow env vars to provide defaults?


i would definitely consider allowing these from the environment.

I agree. I'm wondering what implications would be for different user's setups in the wild. Anything that was not there before is potentially breaking. There is anoter large pending rework of configure on my plate (disabling multithreaded math options, they are useless, basically), so I left this decision till then.

luitjens · 2019-04-09T17:00:32Z

src/configure

+# Define default library roots where known (others may be found by probing).
+case $MATHLIB in
+  MKL) [[ ! $MKLLIBDIR && ! $MKLROOT ]] && MKLROOT=/opt/intel/mkl ;;
+  ATLAS) : ${ATLASROOT:=$(rel2abs ../tools/ATLAS_headers/)} ;;


If we are expecting atlas to be in ../tools/ then we should restore the scripts which download and install it there.

The headers are borked on Ubuntu packages since 16.04, so we supply our own copy. The rest "just works" if you have the libatlas-dev package installed, as I confirmed (see the test log), first two sections.

Please make sure the most common error condition has a clear error message that tells the user the most likely fix.
I expect this most common condition will be when MKL is not installed in /opt/intel/mkl. E.g. maybe encourage the user to try --mathlib=ATLAS or whatever the option is.

In this case, I'd rather encourage the user to install MKL, really. In fact, check_dependencies already does that. There is an argument over OpenBLAS vs MKL on AMD, but ATLAS is a non-contended w.r.t. performance, I believe.

Aha, see this, first heading

Using ATLAS as the linear algebra library. Could not find libatlas.a in any of the generic-Linux places, but we'll try other stuff... ** Failed to configure ATLAS libraries *** ** ERROR ** ** Configure cannot proceed automatically. ** If you know that you have ATLAS installed somewhere on your machine, you ** may be able to proceed by replacing [somewhere] in kaldi.mk with a directory. ** If you have sudo (root) access you could install the ATLAS package on your ** machine, e.g. 'sudo apt-get install libatlas-dev libatlas-base-dev' or ** 'sudo yum install atlas.x86_64' or 'sudo zypper install libatlas3-devel', ** or on cygwin, install atlas from the installer GUI; and then run ./configure ** again. ** ** Otherwise (or if you prefer OpenBLAS for speed), you could go the OpenBLAS ** route: cd to ../tools, type 'extras/install_openblas.sh', cd back to here, ** and type './configure --openblas-root=../tools/OpenBLAS/install'

I think I'd rather s/OpenBLAS/MKL/ here. No objection? OpenBLAS has been a bear to configure recently.

But maybe better as part of a bigger rework #3192. Users are broken now, and this is a hotfix.

luitjens · 2019-04-09T17:05:55Z

src/configure

 #        # support for 64bit ARMv8(AArch64) architecture in Android.

 # This should be incremented after any significant change to the configure
 # script, i.e. any change affecting kaldi.mk or the build system as a whole.


General comment.

We either need to re-add ATLAS support into tools at expected paths or support the canonical paths on different systems. On Ubuntu 16.04 atlas installs to /usr/[lib|include]/atlas but we only allow $ATLASROOT/[lib|include]. We should either let the inc/lib paths be set separately or search both $ATLASROOT/[lib|include] and $ATLASROOT[lib|include]/atlas/

The sad thing is the includes for different algebra libs have been renamed in Ubuntu, like atlas-cblas.h, openblas-cblas.h etc. In particular, atlas has 2 headers (forgot what was the second), and one of them includes the other as <cblas.h>, so they cannot be used as is, without either symlinking or adding our own directory to the end of the compiler's system include path, where a file named cblas.h would be just a one-liner #include <atlas-cblas.h>. The /usr/include/atlas is "internal", the atlas-*.h includes in /usr/include use them, but they are not for the end-user consumption.

14.04 and before did not rename the file cblas.h, but rather declared package conflicts, so you could not have e. g. both libatlas-dev and libopenblas-dev installed at the same time. Now they chose a different route, but introduced a bug. The fact that it's been there unreported and unfixed for may years tells us something about the way people use ATLAS...

I think Dan's approach is the safest: cblas headers do not depend on an implementation¹, so we can carry our own, always possible to #include.

¹ This is not exactly true, as the bitness of ints for row/column indices may vary. But in practice, it's 32 for all packaged BLAS variants, except MKL, where you have a compile/link choice between 32 and 64. Since we are unlikely to go over 4 billion columns or rows, we also use 32-bit indexing with the MKL.

danpovey · 2019-04-09T19:27:06Z

As long as you are confident that this patch would be a clear improvement in ease of installation without breaking anything that works right now, go ahead and merge it.

BTW, @luitjens, the ATLASROOT variable is kind of misleading in the script right now-- we should fix it eventually, maybe in a major rewrite. Right now I think it just points to the location of the headers. But the script does find the system locations of ATLAS quite well as-is.

kkm000 · 2019-04-09T20:14:36Z

Yes, I'll merge, as the users are waiting. I added notes to myself to #3192, please add more if you can think of anything.

Wondering if we should work out an option of a non-root MKL install (either with the installers or by downloading packaged).

I'll test MKL under Windows LSW when I'm bored. I do not see why should not it work, but I'll test. The LSW does not support CUDA, tho, so there's no so much practical value in this currently.

danpovey · 2019-04-09T21:04:21Z

Thanks, yes, a non-root install would be nice if easy, but it seems MKL is very heavyweight with lots of dependencies, so I wonder whether that would be easy to do. A lot of people just want the installation to work and won't care about the exact performance numbers because the most time-consuming thing is the CUDA part anyway. So for me that's a big priority too (easy installation, regardless of performance).

…

On Tue, Apr 9, 2019 at 10:15 AM kkm (aka Kirill Katsnelson) < ***@***.***> wrote: Merged #3216 <#3216> into master. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3216 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADJVu5p64uGlO4LTOp5EtcLHrPxqjtVMks5vfPTJgaJpZM4ck40P> .

kkm000 · 2019-04-09T23:08:04Z

Actually, to me, MKL has always been the easiest option. It does not have any dependencies that are not its own--it is split into multiple packages, true, but it just pulls and installs them all together.

It's 1.8G as set up here (I'm surprised it was only 0.5G on the Mac). I do not really know when the size would be a concern though. Compared to the amounts of data we normally churn it is still not very significant. And CUDA is nearly twice as larger anyway (3.0G for 10.0)

Does CUDA always require root to set up (not including drivers)? I'd think so. At the very least, setting up the per-process GPU mode does.

the most time-consuming thing is the CUDA part anyway

I remember for a full librispeech run it was about half and half in my single-machine configuration. I had a less capable box back then, but only 1 (and slower) GPU, so I do not expect a huge change in the proportion if I did it on the current upgraded machine with 16 AVX512 cores and 2 much better GPUs. On a cluster, the GMM bootstrap part likely parallelizes significantly better that the GPU training.

I'm looking at my logs; training an i-vector extractor on 400hrs (perhaps an overkill) took 80min, this is with MKL and with AVX512 support in CPU; this would be 4hrs with OpenBLAS, If I am to believe those benchmarks (I'll dig up the link); I do not know how much it was optimized for the AVX512 units since then, too. GPU training in the same experiment took some 9hrs. Longer, but of the same order of magnitude. I think there is a reason to make matrix math faster on CPU, and especially for the people who want to start getting real results on a single machine.

[build] Fix configure breakage from kaldi-asr#3194 (MKL default)

d8f412e

Establish MATHLIB as the single ground-truth source for the chosen matrix algebra library.

luitjens reviewed Apr 9, 2019

View reviewed changes

danpovey mentioned this pull request Apr 9, 2019

Cannot set mathlib to ATLAS #3217

Closed

This was referenced Apr 9, 2019

Remove all threaded BLAS options from configure #3192

Closed

Update docs? Intel no longer charges for MKL #3078

Closed

Simplify installation and document the use of Intel MKL #3194

Merged

kkm000 merged commit 4ae4bb0 into kaldi-asr:master Apr 9, 2019

kkm000 mentioned this pull request Apr 10, 2019

Build system #3086

Open

kkm000 deleted the fix-configure-atlas branch April 23, 2019 07:37

[build] Fix configure breakage from #3194 MKL switch #3216

[build] Fix configure breakage from #3194 MKL switch #3216

Uh oh!

Conversation

kkm000 commented Apr 9, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kkm000 commented Apr 9, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Quick tests with MKL and ATLAS

--mathlib=ATLAS, libatlas-dev not installed

Inastall libatlas-dev, use --mathlib=ATLAS

No args, MKL present, ATLAS removed.

Uh oh!

kkm000 commented Apr 9, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kkm000 Apr 9, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danpovey commented Apr 9, 2019

Uh oh!

kkm000 commented Apr 9, 2019

Uh oh!

danpovey commented Apr 9, 2019 via email

Uh oh!

kkm000 commented Apr 9, 2019

Uh oh!

Uh oh!

kkm000 commented Apr 9, 2019 •

edited

Loading

kkm000 commented Apr 9, 2019 •

edited

Loading

kkm000 Apr 9, 2019 •

edited

Loading