5 - onnxruntime_global_thread_pools_test (Failed)

[lvZic]

5: [ OK ] CApiTestGlobalThreadPoolsWithProviders/CApiTestGlobalThreadPoolsWithProvider.simple3/4 (18 ms)
5: [----------] 15 tests from CApiTestGlobalThreadPoolsWithProviders/CApiTestGlobalThreadPoolsWithProvider (1852 ms total)
5:
5: [----------] Global test environment tear-down
5: [==========] 15 tests from 1 test suite ran. (1852 ms total)
5: [ PASSED ] 12 tests.
5: [ FAILED ] 3 tests, listed below:
5: [ FAILED ] CApiTestGlobalThreadPoolsWithProviders/CApiTestGlobalThreadPoolsWithProvider.simple/1, where GetParam() = 1
5: [ FAILED ] CApiTestGlobalThreadPoolsWithProviders/CApiTestGlobalThreadPoolsWithProvider.simple2/1, where GetParam() = 1
5: [ FAILED ] CApiTestGlobalThreadPoolsWithProviders/CApiTestGlobalThreadPoolsWithProvider.simple3/1, where GetParam() = 1
5:
5: 3 FAILED TESTS
5/6 Test #5: onnxruntime_global_thread_pools_test ...***Failed 2.25 sec
test 6
Start 6: onnxruntime_api_tests_without_env

6: Test command: /home/vic/proj/ARVR/onnxruntime/build/Linux/Release/onnxruntime_api_tests_without_env "--gtest_output=xml:/home/vic/proj/ARVR/onnxruntime/build/Linux/Release/onnxruntime_api_tests_without_env.Release.results.xml"
6: Test timeout computed to be: 3600
6: [==========] Running 1 test from 1 test suite.
6: [----------] Global test environment set-up.
6: [----------] 1 test from TestSessionOptions
6: [ RUN ] TestSessionOptions.SetIntraOpNumThreadsWithoutEnv
6: [ OK ] TestSessionOptions.SetIntraOpNumThreadsWithoutEnv (0 ms)
6: [----------] 1 test from TestSessionOptions (0 ms total)
6:
6: [----------] Global test environment tear-down
6: [==========] 1 test from 1 test suite ran. (0 ms total)
6: [ PASSED ] 1 test.
6/6 Test #6: onnxruntime_api_tests_without_env ...... Passed 0.05 sec

83% tests passed, 1 tests failed out of 6

Total Test time (real) = 66.75 sec

The following tests FAILED:
5 - onnxruntime_global_thread_pools_test (Failed)
Errors while running CTest
Output from these tests are in: /home/vic/proj/ARVR/onnxruntime/build/Linux/Release/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
Traceback (most recent call last):
File "/home/vic/proj/ARVR/onnxruntime/tools/ci_build/build.py", line 1986, in
sys.exit(main())
File "/home/vic/proj/ARVR/onnxruntime/tools/ci_build/build.py", line 1924, in main
run_onnxruntime_tests(args, source_dir, ctest_path, build_dir, configs)
File "/home/vic/proj/ARVR/onnxruntime/tools/ci_build/build.py", line 1346, in run_onnxruntime_tests
run_subprocess(ctest_cmd, cwd=cwd, dll_path=dll_path)
File "/home/vic/proj/ARVR/onnxruntime/tools/ci_build/build.py", line 528, in run_subprocess
return run(*args, cwd=cwd, capture_stdout=capture_stdout, shell=shell, env=my_env)
File "/home/vic/proj/ARVR/onnxruntime/tools/python/util/run.py", line 44, in run
env=env, shell=shell)
File "/usr/lib/python3.6/subprocess.py", line 438, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/usr/bin/ctest', '--build-config', 'Release', '--verbose', '--timeout', '3600']' returned non-zero exit status 8.

[lvZic]

env: CUDA Version:11.1 libcudnn8_8.1.0.77-1+cuda11.2_amd64.deb
command: ./build.sh --build_shared_lib --config Release --use_cuda --cudnn_home /usr/local/cuda/ --cuda_home /usr/local/cuda

[fijipants]

I got the same issue on Windows 11, CUDA 11.5.2, cuDNN 8.3.3 (for CUDA 11.5).

Test #1 fails for me too:

1: C:\repo\onnxruntime\onnxruntime\test\providers\provider_test_utils.cc(251): error: The difference between expected[i] and output[i] is 0.0073043107986450195, which exceeds threshold, where
1: expected[i] evaluates to -1.9758384227752686,
1: output[i] evaluates to -1.9831427335739136, and
1: threshold evaluates to 0.004999999888241291.
1: i:1220, provider_type: CUDAExecutionProvider

Steps to reproduce:

• install vs 2019 or 2022
• check out latest or v1.11.0 or v1.10.0 or v1.8.0
• run the following:

.\build.bat --parallel 16 --config Release --build --update --use_cuda --cuda_home="C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5" --cudnn_home="C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5" --cuda_version 11.5
.\build.bat --parallel 16 --config Release --build_nodejs   --use_cuda --cuda_home="C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5" --cudnn_home="C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5" --cuda_version 11.5

NOTE: It works perfectly fine when I remove CUDA:

.\build.bat --parallel 16 --config Release --build --update
.\build.bat --parallel 16 --config Release --build_nodejs

[manickavela29]

facing the similar issue on building in windows , but getParams is initialized to 4.

5: [ FAILED ] CApiTestGlobalThreadPoolsWithProviders/CApiTestGlobalThreadPoolsWithProvider.simple/1, where GetParam() = 4
5: [ FAILED ] CApiTestGlobalThreadPoolsWithProviders/CApiTestGlobalThreadPoolsWithProvider.simple2/1, where GetParam() = 4
5: [ FAILED ] CApiTestGlobalThreadPoolsWithProviders/CApiTestGlobalThreadPoolsWithProvider.simple3/1, where GetParam() = 4

can anyone tell me what i should look out for or what is causing the issue?