Open
Description
Describe the bug
Post-commit CI CUDA E2E test Assert/assert_in_multiple_tus.cpp fails:
FAIL: SYCL :: Assert/assert_in_multiple_tus.cpp (20 of 1270)
******************** TEST 'SYCL :: Assert/assert_in_multiple_tus.cpp' FAILED ********************
Script:
--
: 'RUN: at line 6'; /__w/llvm/llvm/toolchain/bin/clang++ -DSYCL_FALLBACK_ASSERT=1 -fsycl -fsycl-targets=nvptx64-nvidia-cuda -I /__w/llvm/llvm/llvm/sycl/test-e2e/Assert/Inputs /__w/llvm/llvm/llvm/sycl/test-e2e/Assert/assert_in_multiple_tus.cpp /__w/llvm/llvm/llvm/sycl/test-e2e/Assert/Inputs/kernels_in_file2.cpp -o /__w/llvm/llvm/build-e2e/Assert/Output/assert_in_multiple_tus.cpp.tmp.out
: 'RUN: at line 7'; true /__w/llvm/llvm/build-e2e/Assert/Output/assert_in_multiple_tus.cpp.tmp.out &> /__w/llvm/llvm/build-e2e/Assert/Output/assert_in_multiple_tus.cpp.tmp.cpu.txt || true
: 'RUN: at line 8'; true FileCheck /__w/llvm/llvm/llvm/sycl/test-e2e/Assert/assert_in_multiple_tus.cpp --input-file /__w/llvm/llvm/build-e2e/Assert/Output/assert_in_multiple_tus.cpp.tmp.cpu.txt
: 'RUN: at line 9'; env ONEAPI_DEVICE_SELECTOR=ext_oneapi_cuda:gpu SYCL_PI_CUDA_ENABLE_IMAGE_SUPPORT=1 /__w/llvm/llvm/build-e2e/Assert/Output/assert_in_multiple_tus.cpp.tmp.out &> /__w/llvm/llvm/build-e2e/Assert/Output/assert_in_multiple_tus.cpp.tmp.gpu.txt || true
: 'RUN: at line 10'; env ONEAPI_DEVICE_SELECTOR=ext_oneapi_cuda:gpu SYCL_PI_CUDA_ENABLE_IMAGE_SUPPORT=1 FileCheck /__w/llvm/llvm/llvm/sycl/test-e2e/Assert/assert_in_multiple_tus.cpp --input-file /__w/llvm/llvm/build-e2e/Assert/Output/assert_in_multiple_tus.cpp.tmp.gpu.txt
: 'RUN: at line 12'; true /__w/llvm/llvm/build-e2e/Assert/Output/assert_in_multiple_tus.cpp.tmp.out &> /__w/llvm/llvm/build-e2e/Assert/Output/assert_in_multiple_tus.cpp.tmp.acc.txt
: 'RUN: at line 13'; true FileCheck /__w/llvm/llvm/llvm/sycl/test-e2e/Assert/assert_in_multiple_tus.cpp --check-prefix=CHECK-ACC --input-file /__w/llvm/llvm/build-e2e/Assert/Output/assert_in_multiple_tus.cpp.tmp.acc.txt
--
Exit Code: 1
Command Output (stdout):
--
$ ":" "RUN: at line 6"
note: command had no output on stdout or stderr
$ "/__w/llvm/llvm/toolchain/bin/clang++" "-DSYCL_FALLBACK_ASSERT=1" "-fsycl" "-fsycl-targets=nvptx64-nvidia-cuda" "-I" "/__w/llvm/llvm/llvm/sycl/test-e2e/Assert/Inputs" "/__w/llvm/llvm/llvm/sycl/test-e2e/Assert/assert_in_multiple_tus.cpp" "/__w/llvm/llvm/llvm/sycl/test-e2e/Assert/Inputs/kernels_in_file2.cpp" "-o" "/__w/llvm/llvm/build-e2e/Assert/Output/assert_in_multiple_tus.cpp.tmp.out"
# command stderr:
clang++: warning: CUDA version 11.7 is only partially supported [-Wunknown-cuda-version]
$ ":" "RUN: at line 7"
note: command had no output on stdout or stderr
$ "true" "/__w/llvm/llvm/build-e2e/Assert/Output/assert_in_multiple_tus.cpp.tmp.out"
note: command had no output on stdout or stderr
$ ":" "RUN: at line 8"
note: command had no output on stdout or stderr
$ "true" "FileCheck" "/__w/llvm/llvm/llvm/sycl/test-e2e/Assert/assert_in_multiple_tus.cpp" "--input-file" "/__w/llvm/llvm/build-e2e/Assert/Output/assert_in_multiple_tus.cpp.tmp.cpu.txt"
note: command had no output on stdout or stderr
$ ":" "RUN: at line 9"
note: command had no output on stdout or stderr
$ "env" "ONEAPI_DEVICE_SELECTOR=ext_oneapi_cuda:gpu" "SYCL_PI_CUDA_ENABLE_IMAGE_SUPPORT=1" "/__w/llvm/llvm/build-e2e/Assert/Output/assert_in_multiple_tus.cpp.tmp.out"
# redirected output from '/__w/llvm/llvm/build-e2e/Assert/Output/assert_in_multiple_tus.cpp.tmp.gpu.txt':
PI CUDA ERROR:
Value: 710
Name: CUDA_ERROR_ASSERT
Description: device-side assert triggered
Function: build_program
Source Location: /__w/llvm/llvm/src/sycl/plugins/cuda/pi_cuda.cpp:776
PI CUDA ERROR:
Value: 400
Name: CUDA_ERROR_INVALID_HANDLE
Description: invalid resource handle
Function: cuda_piProgramRelease
Source Location: /__w/llvm/llvm/src/sycl/plugins/cuda/pi_cuda.cpp:3600
terminate called after throwing an instance of 'sycl::_V1::compile_program_error'
what(): The program was built for 1 devices
Build program log for 'NVIDIA A10G':
-999 (Unknown PI error)
note: command had no output on stdout or stderr
error: command failed with exit status: -6
$ "true"
note: command had no output on stdout or stderr
$ ":" "RUN: at line 10"
note: command had no output on stdout or stderr
$ "env" "ONEAPI_DEVICE_SELECTOR=ext_oneapi_cuda:gpu" "SYCL_PI_CUDA_ENABLE_IMAGE_SUPPORT=1" "FileCheck" "/__w/llvm/llvm/llvm/sycl/test-e2e/Assert/assert_in_multiple_tus.cpp" "--input-file" "/__w/llvm/llvm/build-e2e/Assert/Output/assert_in_multiple_tus.cpp.tmp.gpu.txt"
# command stderr:
/__w/llvm/llvm/llvm/sycl/test-e2e/Assert/assert_in_multiple_tus.cpp:17:11: error: CHECK: expected string not found in input
// CHECK: {{.*}}kernels_in_file2.cpp:15: int calculus(int): {{global id: \[5|block: \[1}},0,0], {{local id|thread}}: [1,0,0]
^
/__w/llvm/llvm/build-e2e/Assert/Output/assert_in_multiple_tus.cpp.tmp.gpu.txt:1:1: note: scanning from here
^
Input file: /__w/llvm/llvm/build-e2e/Assert/Output/assert_in_multiple_tus.cpp.tmp.gpu.txt
Check file: /__w/llvm/llvm/llvm/sycl/test-e2e/Assert/assert_in_multiple_tus.cpp
-dump-input=help explains the following input dump.
Input was:
<<<<<<
1:
check:17 X error: no match found
2: PI CUDA ERROR:
check:17 ~~~~~~~~~~~~~~~
3: Value: 710
check:17 ~~~~~~~~~~~~
4: Name: CUDA_ERROR_ASSERT
check:17 ~~~~~~~~~~~~~~~~~~~~~~~~~
5: Description: device-side assert triggered
check:17 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
6: Function: build_program
check:17 ~~~~~~~~~~~~~~~~~~~~~~~~~
.
.
.
>>>>>>
error: command failed with exit status: 1
To Reproduce
Run any pre-merge checks on current PRs. It fails in one of my PRs, run history and log files available here: https://github.com/intel/llvm/actions/runs/4542034482/jobs/8005455354?pr=8825
Environment (please complete the following information):
- OS: Linux
- Target device and vendor: AWS Node
- DPC++ version: Latest
Additional context
N/A.