CUDA test hang without GPU.

[xkszltl]

Describe the bug

AllocatorTest.CUDAAllocatorTest hangs forever if build machine does not have GPU available (but with CUDA toolchain installed).
I think in this case CUDA test should be skipped or at least failed, hanging is not the right way.

[----------] 5 tests from AllocatorTest
[ RUN      ] AllocatorTest.CPUAllocatorTest
[       OK ] AllocatorTest.CPUAllocatorTest (0 ms)
[ RUN      ] AllocatorTest.MakeUniquePtrTest
[       OK ] AllocatorTest.MakeUniquePtrTest (0 ms)
[ RUN      ] AllocatorTest.TestOverflowChecks
2020-07-29 10:33:20.7040916 [E:onnxruntime:Default, allocator.cc:28 onnxruntime::IAllocator::CalcMemSizeForArrayWithAlignment] D:\roaster-scratch\onnxruntime\onnxruntime\core/common/safeint.h:17 SafeIntExceptionHandler<class onnxruntime::OnnxRuntimeException>::SafeIntOnOverflow Integer overflow

2020-07-29 10:33:20.7048544 [E:onnxruntime:Default, allocator.cc:28 onnxruntime::IAllocator::CalcMemSizeForArrayWithAlignment] D:\roaster-scratch\onnxruntime\onnxruntime\core/common/safeint.h:17 SafeIntExceptionHandler<class onnxruntime::OnnxRuntimeException>::SafeIntOnOverflow Integer overflow

2020-07-29 10:33:20.7055700 [E:onnxruntime:Default, allocator.cc:28 onnxruntime::IAllocator::CalcMemSizeForArrayWithAlignment] D:\roaster-scratch\onnxruntime\onnxruntime\core/common/safeint.h:17 SafeIntExceptionHandler<class onnxruntime::OnnxRuntimeException>::SafeIntOnOverflow Integer overflow

2020-07-29 10:33:20.7063018 [E:onnxruntime:Default, allocator.cc:28 onnxruntime::IAllocator::CalcMemSizeForArrayWithAlignment] D:\roaster-scratch\onnxruntime\onnxruntime\core/common/safeint.h:17 SafeIntExceptionHandler<class onnxruntime::OnnxRuntimeException>::SafeIntOnOverflow Integer overflow

[       OK ] AllocatorTest.TestOverflowChecks (3 ms)
[ RUN      ] AllocatorTest.CUDAAllocatorTest

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Win Server 2019
• ONNX Runtime installed from (source or binary): source
• ONNX Runtime version: 1.4.0
• Python version: 3.8.5
• Visual Studio version (if applicable): 2019
• CUDA/cuDNN version: 11.0 (10.2 also have this issue)
• GPU model and memory: N/A

[skottmckay]

Isn't this somewhat equivalent to cross-compiling? You're building something that can't run on the current machine.

Is there a viable way to detect this scenario from a unit test?
Even if there is a way, having to update lots of tests to handle this sort of thing doesn't really add value to the product.

You can skip the tests by passing the --skip_tests flag to the build script.

[xkszltl]

We do rely on unit test to make sure our build quality is good and no missing dll dependency.

I understand your point, but we don't want to skip CPU test as well when building CUDA.
Since GPU is pluggable device, should any CUDA test start with a cudaGetDevice(), also to set the correct device id?

[xkszltl]

Also, to clarify, the problematic part is not "it doesn't work", but "it doesn't quit".
IMO even segfault is a test result.

[snnn]

So, you're saying, you build onnxruntime with CUDA, then run it on a machine with CUDA installed, but without Nvidia driver because the machine doesn't have the hardware, and you expect when you run the tests they should pass?

[xkszltl]

No, I said they can fail, but not hang.

[skottmckay]

I did a build on a VM with no CUDA device (Ubuntu 18.04) and the test fails for me with a reasonable error message. Not sure why it hangs for you.

I tested with CUDA 10.2 with CUDNN 7.6.5 installed.

[ RUN ] AllocatorTest.CUDAAllocatorTest
unknown file: Failure
C++ exception with description "/home/scott/src/github/onnxruntime/onnxruntime/core/providers/cuda/cuda_call.cc:123 bool onnxruntime::CudaCall(ERRTYPE, const char*, const char*, ERRTYPE, const char*) [with ERRTYPE = cudaError; bool THRW = true] /home/scott/src/github/onnxruntime/onnxruntime/core/providers/cuda/cuda_call.cc:117 bool onnxruntime::CudaCall(ERRTYPE, const char*, const char*, ERRTYPE, const char*) [with ERRTYPE = cudaError; bool THRW = true] CUDA failure 100: no CUDA-capable device is detected ; GPU=0 ; hostname=Ubuntu1804 ; expr=cuda_err;

[xkszltl]

Are you testing master or 1.4.0?
On Windows I get this from other tests, but not AllocatorTest.CUDAAllocatorTest

Provider:CUDAExecutionProvider
unknown file: error: C++ exception with description "D:\roaster-scratch\onnxruntime\onnxruntime\core\providers\cuda\cuda_call.cc:123 onnxruntime::CudaCall D:\roaster-scratch\onnxruntime\onnxruntime\core\providers\cuda\cuda_call.cc:117 onnxruntime::CudaCall CUDA failure 35: CUDA driver version is insufficient for CUDA runtime version ; GPU=-679215104 ; hostname=Roaster-Win ; expr=cudaSetDevice(device_id_);

" thrown in the test body.

[xkszltl]

And that's the only one hang.

[skottmckay]

Are you able to step through AllocatorTest.CUDAAllocatorTest in a debugger (can be done via Test Explorer) and identify the call where it hangs?

[xkszltl]

Not really....RDP experience from China to Azure VM is too bad recently...

[xkszltl]

@skottmckay
I got a stack trace on Linux about where is hang:

root@64e141d47b68:/tmp/scratch/onnxruntime/build#  gdb --ex 'r' --args ./onnxruntime_test_all --gtest_filter='AllocatorTest.CUDAAllocatorTest'
GNU gdb (Ubuntu 8.1-0ubuntu3.2) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./onnxruntime_test_all...done.
Starting program: /tmp/scratch/onnxruntime/build/onnxruntime_test_all --gtest_filter=AllocatorTest.CUDAAllocatorTest
warning: Error disabling address space randomization: Operation not permitted
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Note: Google Test filter = AllocatorTest.CUDAAllocatorTest
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from AllocatorTest
[ RUN      ] AllocatorTest.CUDAAllocatorTest
^C
Program received signal SIGINT, Interrupt.
0x00007fda9d3ab412 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
(gdb) bt
#0  0x00007fda9d3ab412 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#1  0x00007fda9d3ab469 in std::locale::locale() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x00007fda9d3a8d8b in std::ios_base::_M_init() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x00007fda9d3fdf11 in std::basic_ios<char, std::char_traits<char> >::init(std::basic_streambuf<char, std::char_traits<char> >*) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x000055b05832fc7e in std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::basic_ostringstream (__mode=std::_S_out, this=0x7fffdaa6b580, __in_chrg=<optimized out>, __vtt_parm=<optimized out>)
    at /usr/include/c++/8/bits/char_traits.h:287
#5  onnxruntime::OnnxRuntimeException::OnnxRuntimeException (this=0x55b05e561ea0, location=..., failed_condition=0x0, 
    msg="../onnxruntime/core/providers/cuda/cuda_call.cc:117 bool onnxruntime::CudaCall(ERRTYPE, const char*, const char*, ERRTYPE, const char*) [with ERRTYPE = cudaError; bool THRW = true] CUDA failure 35: CU"...)
    at ../include/onnxruntime/core/common/exceptions.h:44
#6  0x000055b05894cf55 in onnxruntime::OnnxRuntimeException::OnnxRuntimeException (
    msg="../onnxruntime/core/providers/cuda/cuda_call.cc:117 bool onnxruntime::CudaCall(ERRTYPE, const char*, const char*, ERRTYPE, const char*) [with ERRTYPE = cudaError; bool THRW = true] CUDA failure 35: CU"..., location=..., this=0x55b05e561ea0)
    at ../include/onnxruntime/core/common/exceptions.h:31
#7  onnxruntime::CudaCall<cudaError, true> (retCode=<optimized out>, exprString=exprString@entry=0x55b05959bb71 "cudaMalloc((void**)&p, size)", libName=libName@entry=0x55b0594bcfde "CUDA", successCode=successCode@entry=cudaSuccess, 
    msg=msg@entry=0x55b059671b29 "") at ../onnxruntime/core/providers/cuda/cuda_call.cc:123
#8  0x000055b058949209 in onnxruntime::CUDAAllocator::Alloc (this=<optimized out>, size=2304) at ../onnxruntime/core/providers/cuda/cuda_allocator.cc:40
#9  0x000055b0591ded66 in onnxruntime::BFCArena::<lambda(size_t)>::operator() (__closure=<synthetic pointer>, alloc_bytes=2304) at /usr/include/c++/8/bits/unique_ptr.h:345
#10 onnxruntime::BFCArena::Extend(unsigned long) () at ../onnxruntime/core/framework/bfc_arena.cc:140
#11 0x000055b0591e47ef in onnxruntime::BFCArena::AllocateRawInternal(unsigned long, bool) () at ../onnxruntime/core/framework/bfc_arena.cc:267
#12 0x000055b05842a3ed in onnxruntime::test::AllocatorTest_CUDAAllocatorTest_Test::TestBody() () at /usr/include/c++/8/bits/shared_ptr_base.h:1018
#13 0x00007fda9db7452a in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (location=0x7fda9db75992 "the test body", method=<optimized out>, object=0x55b05e55dd00) at ../googletest/src/gtest.cc:2414
#14 testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=object@entry=0x55b05e55dd00, method=<optimized out>, location=location@entry=0x7fda9db75992 "the test body") at ../googletest/src/gtest.cc:2469
#15 0x00007fda9db6a3a3 in testing::Test::Run (this=0x55b05e55dd00) at ../googletest/src/gtest.cc:2508
#16 testing::Test::Run (this=0x55b05e55dd00) at ../googletest/src/gtest.cc:2498
#17 0x00007fda9db6a505 in testing::TestInfo::Run (this=0x55b05e23cfd0) at ../googletest/src/gtest.cc:2684
#18 testing::TestInfo::Run (this=0x55b05e23cfd0) at ../googletest/src/gtest.cc:2657
#19 0x00007fda9db6a5e5 in testing::TestSuite::Run (this=0x55b05e23bb60) at ../googletest/src/gtest.cc:2816
#20 testing::TestSuite::Run (this=0x55b05e23bb60) at ../googletest/src/gtest.cc:2795
#21 0x00007fda9db6ab03 in testing::internal::UnitTestImpl::RunAllTests() () at /usr/include/c++/8/bits/stl_vector.h:930
#22 0x00007fda9db6ad21 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (location=0x7fda9db77be0 "auxiliary test code (environments or event listeners)", method=<optimized out>, object=0x55b05e21e1f0)
    at ../googletest/src/gtest.cc:2414
#23 testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (location=0x7fda9db77be0 "auxiliary test code (environments or event listeners)", 
    method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x7fda9db6a680 <testing::internal::UnitTestImpl::RunAllTests()>, object=0x55b05e21e1f0) at ../googletest/src/gtest.cc:2469
#24 testing::UnitTest::Run() () at ../googletest/src/gtest.cc:4925
#25 0x000055b0582f00e6 in RUN_ALL_TESTS () at /usr/local/include/gtest/gtest.h:2473
#26 main () at ../onnxruntime/test/providers/test_main.cc:50
#27 0x00007fda9c97fb97 in __libc_start_main (main=0x55b0582f0060 <main>, argc=2, argv=0x7fffdaa6c5d8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffdaa6c5c8) at ../csu/libc-start.c:310
#28 0x000055b0583130fa in _start () at /usr/include/c++/8/bits/char_traits.h:322

And from cuda-gdb:

root@64e141d47b68:/tmp/scratch/onnxruntime/build# cuda-gdb --ex 'r' --args ./onnxruntime_test_all --gtest_filter='AllocatorTest.CUDAAllocatorTest'
NVIDIA (R) CUDA Debugger
11.0 release
Portions Copyright (C) 2007-2020 NVIDIA Corporation
GNU gdb (GDB) 8.2
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./onnxruntime_test_all...done.
Starting program: /tmp/scratch/onnxruntime/build/onnxruntime_test_all --gtest_filter=AllocatorTest.CUDAAllocatorTest
warning: Error disabling address space randomization: Operation not permitted
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Note: Google Test filter = AllocatorTest.CUDAAllocatorTest
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from AllocatorTest
[ RUN      ] AllocatorTest.CUDAAllocatorTest
^C
Program received signal SIGINT, Interrupt.
0x00007f5324ab7f28 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
(cuda-gdb) bt
#0  0x00007f5324ab7f28 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f53253fa51b in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::find(char const*, unsigned long, unsigned long) const () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x000055586901bb58 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::find (__pos=0, __s=0x55586a4472c5 "cudaMalloc", this=0x7fff23fba8a0) at /usr/include/c++/8/bits/char_traits.h:322
#3  onnxruntime::BFCArena::<lambda(size_t)>::operator() (__closure=<synthetic pointer>, alloc_bytes=2304) at ../onnxruntime/core/framework/bfc_arena.cc:90
#4  onnxruntime::BFCArena::Extend (this=0x55586fedf7f0, rounded_bytes=1024) at ../onnxruntime/core/framework/bfc_arena.cc:140
#5  0x0000555869fd77ef in onnxruntime::BFCArena::AllocateRawInternal (this=0x55586fedf7f0, num_bytes=1024, dump_log_on_failure=<optimized out>) at ../onnxruntime/core/framework/bfc_arena.cc:267
#6  0x000055586921d3ed in onnxruntime::test::AllocatorTest_CUDAAllocatorTest_Test::TestBody (this=<optimized out>) at /usr/include/c++/8/bits/shared_ptr_base.h:1018
#7  0x00007f5325b4352a in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (location=0x7f5325b44992 "the test body", method=<optimized out>, object=0x55586fedbd00) at ../googletest/src/gtest.cc:2414
#8  testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=object@entry=0x55586fedbd00, method=<optimized out>, location=location@entry=0x7f5325b44992 "the test body") at ../googletest/src/gtest.cc:2469
#9  0x00007f5325b393a3 in testing::Test::Run (this=0x55586fedbd00) at ../googletest/src/gtest.cc:2508
#10 testing::Test::Run (this=0x55586fedbd00) at ../googletest/src/gtest.cc:2498
#11 0x00007f5325b39505 in testing::TestInfo::Run (this=0x55586fbbafd0) at ../googletest/src/gtest.cc:2684
#12 testing::TestInfo::Run (this=0x55586fbbafd0) at ../googletest/src/gtest.cc:2657
#13 0x00007f5325b395e5 in testing::TestSuite::Run (this=0x55586fbb9b60) at ../googletest/src/gtest.cc:2816
#14 testing::TestSuite::Run (this=0x55586fbb9b60) at ../googletest/src/gtest.cc:2795
#15 0x00007f5325b39b03 in testing::internal::UnitTestImpl::RunAllTests (this=0x55586fb9c1f0) at /usr/include/c++/8/bits/stl_vector.h:930
#16 0x00007f5325b39d21 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (location=0x7f5325b46be0 "auxiliary test code (environments or event listeners)", method=<optimized out>, object=0x55586fb9c1f0)
    at ../googletest/src/gtest.cc:2414
#17 testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (location=0x7f5325b46be0 "auxiliary test code (environments or event listeners)", 
    method=(bool (testing::internal::UnitTestImpl::*)(class testing::internal::UnitTestImpl * const)) 0x7f5325b39680 <testing::internal::UnitTestImpl::RunAllTests()>, object=0x55586fb9c1f0) at ../googletest/src/gtest.cc:2469
#18 testing::UnitTest::Run (this=0x7f5325d58100 <testing::UnitTest::GetInstance()::instance>) at ../googletest/src/gtest.cc:4925
#19 0x00005558690e30e6 in RUN_ALL_TESTS () at /usr/local/include/gtest/gtest.h:2473
#20 main (argc=<optimized out>, argv=<optimized out>) at ../onnxruntime/test/providers/test_main.cc:50
(cuda-gdb) 

It ends up at different places every time so probably something is slowly moving forward (or trapped in a infinite loop).

[xkszltl]

Here's where it is after several hours:

#0  0x00007fa29f8af87d in ?? () from /lib/x86_64-linux-gnu/libgcc_s.so.1
#1  0x00007fa29f8b0440 in ?? () from /lib/x86_64-linux-gnu/libgcc_s.so.1
#2  0x00007fa29f8b198b in _Unwind_RaiseException () from /lib/x86_64-linux-gnu/libgcc_s.so.1
#3  0x00007fa29fee9d47 in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00005601359a5f8e in onnxruntime::CudaCall<cudaError, true> (retCode=<optimized out>, exprString=exprString@entry=0x5601365f4b71 "cudaMalloc((void**)&p, size)", libName=libName@entry=0x560136515fde "CUDA", 
    successCode=successCode@entry=cudaSuccess, msg=msg@entry=0x5601366cab29 "") at /usr/include/c++/8/ext/new_allocator.h:86
#5  0x00005601359a2209 in onnxruntime::CUDAAllocator::Alloc (this=<optimized out>, size=2304) at ../onnxruntime/core/providers/cuda/cuda_allocator.cc:40
#6  0x0000560136237d66 in onnxruntime::BFCArena::<lambda(size_t)>::operator() (__closure=<synthetic pointer>, alloc_bytes=2304) at /usr/include/c++/8/bits/unique_ptr.h:345
#7  onnxruntime::BFCArena::Extend (this=0x56013b1a37f0, rounded_bytes=1024) at ../onnxruntime/core/framework/bfc_arena.cc:140
#8  0x000056013623d7ef in onnxruntime::BFCArena::AllocateRawInternal (this=0x56013b1a37f0, num_bytes=1024, dump_log_on_failure=<optimized out>) at ../onnxruntime/core/framework/bfc_arena.cc:267
#9  0x00005601354833ed in onnxruntime::test::AllocatorTest_CUDAAllocatorTest_Test::TestBody (this=<optimized out>) at /usr/include/c++/8/bits/shared_ptr_base.h:1018
#10 0x00007fa2a06c652a in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (location=0x7fa2a06c7992 "the test body", method=<optimized out>, object=0x56013b19fd00) at ../googletest/src/gtest.cc:2414
#11 testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=object@entry=0x56013b19fd00, method=<optimized out>, location=location@entry=0x7fa2a06c7992 "the test body") at ../googletest/src/gtest.cc:2469
#12 0x00007fa2a06bc3a3 in testing::Test::Run (this=0x56013b19fd00) at ../googletest/src/gtest.cc:2508
#13 testing::Test::Run (this=0x56013b19fd00) at ../googletest/src/gtest.cc:2498
#14 0x00007fa2a06bc505 in testing::TestInfo::Run (this=0x56013ae7efd0) at ../googletest/src/gtest.cc:2684
#15 testing::TestInfo::Run (this=0x56013ae7efd0) at ../googletest/src/gtest.cc:2657
#16 0x00007fa2a06bc5e5 in testing::TestSuite::Run (this=0x56013ae7db60) at ../googletest/src/gtest.cc:2816
#17 testing::TestSuite::Run (this=0x56013ae7db60) at ../googletest/src/gtest.cc:2795
#18 0x00007fa2a06bcb03 in testing::internal::UnitTestImpl::RunAllTests (this=0x56013ae601f0) at /usr/include/c++/8/bits/stl_vector.h:930
#19 0x00007fa2a06bcd21 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (location=0x7fa2a06c9be0 "auxiliary test code (environments or event listeners)", method=<optimized out>, object=0x56013ae601f0)
    at ../googletest/src/gtest.cc:2414
#20 testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (location=0x7fa2a06c9be0 "auxiliary test code (environments or event listeners)", 
    method=(bool (testing::internal::UnitTestImpl::*)(class testing::internal::UnitTestImpl * const)) 0x7fa2a06bc680 <testing::internal::UnitTestImpl::RunAllTests()>, object=0x56013ae601f0) at ../googletest/src/gtest.cc:2469
#21 testing::UnitTest::Run (this=0x7fa2a08db100 <testing::UnitTest::GetInstance()::instance>) at ../googletest/src/gtest.cc:4925
#22 0x00005601353490e6 in RUN_ALL_TESTS () at /usr/local/include/gtest/gtest.h:2473
#23 main (argc=<optimized out>, argv=<optimized out>) at ../onnxruntime/test/providers/test_main.cc:50