Failed OpenMP assertion when loading MXNet compiled with DEBUG=1

[anirudh2290]

Reproduction steps:

cmake -DUSE_CUDA=ON -DUSE_CUDNN=ON -DUSE_MKLDNN=ON -DBLAS=Open -DUSE_DEBUG=1 -DCMAKE_BUILD_TYPE=Debug -GNinja .. && ninja && cd ..

[INFO] Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=1107463865 to reproduce.
Test should successfully run its course of multi-process/forked data loader without errors ... Assertion failure at kmp_runtime.cpp(6481): __kmp_team_pool == __null.
OMP: Error #13: Assertion failure at kmp_runtime.cpp(6481).
OMP: Hint: Please submit a bug report with this message, compile and run commands used, and machine configuration info including native compiler and operating system versions. Faster response will be obtained by including all program sources. For information on submitting this issue, please see https://bugs.llvm.org/.
Assertion failure at kmp_runtime.cpp(6481): __kmp_team_pool == __null.
OMP: Error #13: Assertion failure at kmp_runtime.cpp(6481).
OMP: Hint: Please submit a bug report with this message, compile and run commands used, and machine configuration info including native compiler and operating system versions. Faster response will be obtained by including all program sources. For information on submitting this issue, please see https://bugs.llvm.org/.
^C[INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=39684479 to reproduce.

----------------------------------------------------------------------
Ran 1 test in 2.106s

OK

----------System Info----------
('Platform     :', 'Linux-4.4.0-1055-aws-x86_64-with-Ubuntu-16.04-xenial')
('system       :', 'Linux')
('node         :', 'ip-172-31-21-146')
('release      :', '4.4.0-1055-aws')
('version      :', '#64-Ubuntu SMP Thu Apr 5 17:06:36 UTC 2018')
----------Hardware Info----------
('machine      :', 'x86_64')
('processor    :', 'x86_64')
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                32
On-line CPU(s) list:   0-31
Thread(s) per core:    2
Core(s) per socket:    16
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 79
Model name:            Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz
Stepping:              1
CPU MHz:               2699.984
CPU max MHz:           3000.0000
CPU min MHz:           1200.0000
BogoMIPS:              4600.18
Hypervisor vendor:     Xen
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              46080K
NUMA node0 CPU(s):     0-31
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single retpoline kaiser fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx xsaveopt

[KellenSunderland]

I'd suggest tweaking this description slightly. Instead of "test_multi_worker_forked_data_loader fails with DEBUG=" I'd suggest "Failed OpenMP assertion when loading MXNet compiled with DEBUG=1".

It's happening for me (and Pedro) in non-dataloading scenarios. Seems to happen as soon as the thread pool is created.

[cjolivier01]

I don't see a root cause listed here.

[cjolivier01]

So, why is __kmp_team_pool == __null assertion hitting? Intel OpenMP is compatible with process forking, so this is probably a symptom of something else wrong leading up to this.

[larroy]

This is really problematic, is there a suggestion for anything that we can do to fix this?

[larroy]

Stuck for 46h running unit tests with the following error message:

OMP: Error #13: Assertion failure at kmp_runtime.cpp(6479).
OMP: Hint: Please submit a bug report with this message, compile and run commands used, and machine configuration info including native compiler and operating system versions. Faster response will be obtained by including all program sources. For information on submitting this issue, please see https://bugs.llvm.org/.
Assertion failure at kmp_runtime.cpp(6479): __kmp_thread_pool == __null.

[larroy]

I spent some time looking at this, I can say I understand what's the problem, this happens when the omp engine is initialized after destructors are called on thread creation. Usually involves the pthread_at_fork fiasco in initialize.cc

Iinitalization is triggered from __nptl_deallocate_tsd (); https://code.woboq.org/userspace/glibc/nptl/pthread_create.c.html#497

__kmp_team_pool is a volatile which gets changed continously across threads for reuse of the last kmp team.

This happens if omp is initialized twice for some reason, so __kmp_team_pool is not NULL because there's a thread running omp functions, kmp_team_pool can be non-NULL

[larroy]

What's worse is that when executing this test, you can easily create a segmentation fault, by hitting ^C so we are UB land:

python tests/python/unittest/test_gluon_data.py

OMP: Error #13: Assertion failure at kmp_runtime.cpp(6479).
OMP: Hint: Please submit a bug report with this message, compile and run commands used, and machine configuration info including native compiler and operating system versions. Faster response will be obtained by including all program sources. For information on submitting this issue, please see https://bugs.llvm.org/.
Assertion failure at kmp_runtime.cpp(6479): __kmp_thread_pool == __null.
OMP: Error #13: Assertion failure at kmp_runtime.cpp(6479).
OMP: Hint: Please submit a bug report with this message, compile and run commands used, and machine configuration info including native compiler and operating system versions. Faster response will be obtained by including all program sources. For information on submitting this issue, please see https://bugs.llvm.org/.
Assertion failure at kmp_runtime.cpp(6479): __kmp_thread_pool == __null.
OMP: Error #13: Assertion failure at kmp_runtime.cpp(6479).
OMP: Hint: Please submit a bug report with this message, compile and run commands used, and machine configuration info including native compiler and operating system versions. Faster response will be obtained by including all program sources. For information on submitting this issue, please see https://bugs.llvm.org/.
^C[INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1709269671 to reproduce.

----------------------------------------------------------------------
Ran 3 tests in 12.296s

OK

Segmentation fault: 11

Stack trace returned 10 entries:
[bt] (0) /home/piotr/devel/mxnet/mxnet/python/mxnet/../../build/libmxnet.so(dmlc::StackTrace[abi:cxx11]()+0x54) [0x7fef667f38e5]
[bt] (1) /home/piotr/devel/mxnet/mxnet/python/mxnet/../../build/libmxnet.so(+0x4ba60be) [0x7fef69ba40be]
[bt] (2) /lib/x86_64-linux-gnu/libc.so.6(+0x354b0) [0x7fefbeb324b0]
[bt] (3) /home/piotr/devel/mxnet/mxnet/build/3rdparty/openmp/runtime/src/libomp.so(+0xc7c0d) [0x7fefb8857c0d]
[bt] (4) /home/piotr/devel/mxnet/mxnet/build/3rdparty/openmp/runtime/src/libomp.so(+0xc7d02) [0x7fefb8857d02]
[bt] (5) /home/piotr/devel/mxnet/mxnet/build/3rdparty/openmp/runtime/src/libomp.so(+0x6ebca) [0x7fefb87febca]
[bt] (6) /home/piotr/devel/mxnet/mxnet/build/3rdparty/openmp/runtime/src/libomp.so(+0x56e9d) [0x7fefb87e6e9d]
[bt] (7) /home/piotr/devel/mxnet/mxnet/build/3rdparty/openmp/runtime/src/libomp.so(+0x5727c) [0x7fefb87e727c]
[bt] (8) /home/piotr/devel/mxnet/mxnet/build/3rdparty/openmp/runtime/src/libomp.so(+0x576b4) [0x7fefb87e76b4]
[bt] (9) /home/piotr/devel/mxnet/mxnet/build/3rdparty/openmp/runtime/src/libomp.so(+0x56d4f) [0x7fefb87e6d4f]

[piyushghai]

This is not related to Data-loading, but is an issue with OpenMP. Removing the data-loading label.

@mxnet-label-bot Update [Backend, Bug]