Sporadic llvm-test-suite failures on CUDA back-end.

[bader]

@jchlanda, do you think that this failure on NVIDIA GPU https://github.com/intel/llvm/runs/7391421134?check_suite_focus=true is related to this change? This change is not expected to change the default behavior.

It doesn't look related, even if we ignore the assert messages which are a bit flaky, upon hitting the assert PI runtime should generate error 710:

PI CUDA ERROR:
        Value:           710
        Name:            CUDA_ERROR_ASSERT

but this one reports -999 - Unknown PI error (https://github.com/intel/llvm/runs/7391421134?check_suite_focus=true#step:8:1168)

I can't reproduce this issue locally, so I restarted CI jobs and it failed again, but with the different error:

error: command failed with exit status: 255

https://github.com/intel/llvm/runs/7450227286?check_suite_focus=true

These look like flaky llvm-test-suite issues not caused by this patch. I'll file a report to investigate them and merge this PR.

Originally posted by @bader in #6378 (comment)

Update: archives with logs logs_46239.zip
logs_46467.zip

[AlexeySachkov]

It seems like I've encountered the same issue here. Archived logs: logs_47906.zip:

$ "env" "SYCL_PI_LEVEL_ZERO_TRACK_INDIRECT_ACCESS_MEMORY=1" "SYCL_PI_SUPPRESS_ERROR_MESSAGE=1" "env" "SYCL_DEVICE_FILTER=ext_oneapi_cuda:gpu,host" "SYCL_PI_CUDA_ENABLE_IMAGE_SUPPORT=1" "/__w/llvm/llvm/build/SYCL/Assert/Output/assert_in_simultaneous_kernels.cpp.tmp.out"
# redirected output from '/__w/llvm/llvm/build/SYCL/Assert/Output/assert_in_simultaneous_kernels.cpp.tmp.txt':
terminate called after throwing an instance of 'cl::sycl::runtime_error'
  what():  Native API failed. Native API returns: -999 (Unknown PI error) -999 (Unknown PI error)
  
$ "env" "SYCL_PI_LEVEL_ZERO_TRACK_INDIRECT_ACCESS_MEMORY=1" "SYCL_PI_SUPPRESS_ERROR_MESSAGE=1" "env" "SYCL_DEVICE_FILTER=ext_oneapi_cuda:gpu,host" "SYCL_PI_CUDA_ENABLE_IMAGE_SUPPORT=1" "/__w/llvm/llvm/build/SYCL/Assert/Output/assert_in_simultaneously_multiple_tus.cpp.tmp.out"
# redirected output from '/__w/llvm/llvm/build/SYCL/Assert/Output/assert_in_simultaneously_multiple_tus.cpp.tmp.txt':
terminate called after throwing an instance of 'cl::sycl::runtime_error'
  what():  Native API failed. Native API returns: -999 (Unknown PI error) -999 (Unknown PI error)

I doubt that those are regressions caused by my changes (if that was the case, I would expect all assert tests to fail, not just two random ones)

[bader]

It fails for #6439 here https://github.com/intel/llvm/runs/7764332078?check_suite_focus=true as well.
Probably we should disable execution of these tests and prioritize investigation if happens frequently.

[JackAKirk]

I worked out that at least one of the 999 unknown errors must be CUDA_ERROR_NOT_INITIALIZED and the failures may have originated with the changes in #6296 which could prevent a shared cuda context from being initialized for each thread that uses it. Investigation still ongoing.

[romanovvlad]

Disabled one more, probably related, tests assert_in_simultaneously_multiple_tus_one_ndebug.cpp here: intel/llvm-test-suite#1186