SYCL::Plugin/level_zero_batch_event_status.cpp fails sporadically

[alexbatashev]

Log excerpt:

FAIL: SYCL :: Plugin/level_zero_batch_event_status.cpp (597 of 807)
******************** TEST 'SYCL :: Plugin/level_zero_batch_event_status.cpp' FAILED ********************
Script:
--
: 'RUN: at line 3';    /__w/llvm/llvm/toolchain/bin/clang++      -fsycl -fsycl-unnamed-lambda -fsycl-targets=spir64  /__w/llvm/llvm/llvm_test_suite/SYCL/Plugin/level_zero_batch_event_status.cpp -o /__w/llvm/llvm/build/SYCL/Plugin/Output/level_zero_batch_event_status.cpp.tmp.out
: 'RUN: at line 6';   env SYCL_PI_LEVEL_ZERO_BATCH_SIZE=4 SYCL_PI_TRACE=2 ZE_DEBUG=1  env SYCL_DEVICE_FILTER=ext_oneapi_level_zero:gpu,host  /__w/llvm/llvm/build/SYCL/Plugin/Output/level_zero_batch_event_status.cpp.tmp.out 2>&1 | FileCheck /__w/llvm/llvm/llvm_test_suite/SYCL/Plugin/level_zero_batch_event_status.cpp
--
Exit Code: 1

Command Output (stdout):
--
$ ":" "RUN: at line 3"
note: command had no output on stdout or stderr
$ "/__w/llvm/llvm/toolchain/bin/clang++" "-fsycl" "-fsycl-unnamed-lambda" "-fsycl-targets=spir64" "/__w/llvm/llvm/llvm_test_suite/SYCL/Plugin/level_zero_batch_event_status.cpp" "-o" "/__w/llvm/llvm/build/SYCL/Plugin/Output/level_zero_batch_event_status.cpp.tmp.out"
note: command had no output on stdout or stderr
$ ":" "RUN: at line 6"
note: command had no output on stdout or stderr
$ "env" "SYCL_PI_LEVEL_ZERO_BATCH_SIZE=4" "SYCL_PI_TRACE=2" "ZE_DEBUG=1" "env" "SYCL_DEVICE_FILTER=ext_oneapi_level_zero:gpu,host" "/__w/llvm/llvm/build/SYCL/Plugin/Output/level_zero_batch_event_status.cpp.tmp.out"
note: command had no output on stdout or stderr
$ "FileCheck" "/__w/llvm/llvm/llvm_test_suite/SYCL/Plugin/level_zero_batch_event_status.cpp"
# command stderr:
/__w/llvm/llvm/llvm_test_suite/SYCL/Plugin/level_zero_batch_event_status.cpp:29:11: error: CHECK: expected string not found in input
// CHECK: ---> piEnqueueKernelLaunch
          ^
<stdin>:447:20: note: scanning from here
---> piEventGetInfo(
                   ^
<stdin>:468:1: note: possible intended match here
---> piQueueRelease(
^

Input file: <stdin>
Check file: /__w/llvm/llvm/llvm_test_suite/SYCL/Plugin/level_zero_batch_event_status.cpp

-dump-input=help explains the following input dump.

Input was:
<<<<<<
            .
            .
            .
          442: ZE ---> zeCommandListClose(CommandList->first) 
          443: ZE ---> zeCommandQueueExecuteCommandLists(ZeCommandQueue, 1, &ZeCommandList, CommandList->second.ZeFence) 
          444: ZE ---> zeHostSynchronize(ZeQueue) 
          445: ) ---> pi_result : PI_SUCCESS 
          446:  
          447: ---> piEventGetInfo( 
check:29'0                        X~ error: no match found
          448:  pi_event : 0x1bee7d0 
check:29'0     ~~~~~~~~~~~~~~~~~~~~~~
          449:  <unknown> : 4563 
check:29'0     ~~~~~~~~~~~~~~~~~~
          450:  <unknown> : 4 
check:29'0     ~~~~~~~~~~~~~~~
          451:  <unknown> : 0x7ffcc900b58c 
check:29'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          452:  <nullptr> 
check:29'0     ~~~~~~~~~~~
            .
            .
            .
          463:  pi_event : 0x1d12140 
check:29'0     ~~~~~~~~~~~~~~~~~~~~~~
          464: ZE ---> zeEventDestroy(Event->ZeEvent) 
check:29'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          465: PI ---> QueueRelease(Event->Queue, LockedQueue) 
check:29'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          466: ) ---> pi_result : PI_SUCCESS 
check:29'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          467:  
check:29'0     ~
          468: ---> piQueueRelease( 
check:29'0     ~~~~~~~~~~~~~~~~~~~~~
check:29'1     ?                     possible intended match
          469:  <unknown> : 0x1bfbd00 
check:29'0     ~~~~~~~~~~~~~~~~~~~~~~~
          470: ZE ---> zeHostSynchronize(ZeQueue) 
check:29'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          471: ZE ---> zeFenceReset(CommandList->second.ZeFence) 
check:29'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          472: ZE ---> zeCommandListReset(CommandList->first) 
check:29'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          473: PI ---> piKernelRelease(pi_cast<pi_kernel>(CommandData)) 
check:29'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            .
            .
            .
>>>>>>

error: command failed with exit status: 1

--

Driver info:

  [opencl:gpu:0] Intel(R) OpenCL HD Graphics, Intel(R) UHD Graphics 630 [0x3e92] 3.0 [22.09.22577]
  [opencl:cpu:1] Intel(R) OpenCL, Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz 3.0 [2021.13.11.0.23_160000]
  [ext_oneapi_level_zero:gpu:0] Intel(R) Level-Zero, Intel(R) UHD Graphics 630 [0x3e92] 1.3 [1.3.22577]
  [host:host:0] SYCL host platform, SYCL host device 1.2 [1.2]
  
  Platforms: 4
  Platform [#1]:
      Version  : OpenCL 3.0 
      Name     : Intel(R) OpenCL HD Graphics
      Vendor   : Intel(R) Corporation
      Devices  : 1
          Device [#0]:
          Type       : gpu
          Version    : 3.0
          Name       : Intel(R) UHD Graphics 630 [0x3e92]
          Vendor     : Intel(R) Corporation
          Driver     : 22.09.22577
  Platform [#2]:
      Version  : OpenCL 3.0 LINUX
      Name     : Intel(R) OpenCL
      Vendor   : Intel(R) Corporation
      Devices  : 1
          Device [#1]:
          Type       : cpu
          Version    : 3.0
          Name       : Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
          Vendor     : Intel(R) Corporation
          Driver     : 2021.13.11.0.23_160000
  Platform [#3]:
      Version  : 1.3
      Name     : Intel(R) Level-Zero
      Vendor   : Intel(R) Corporation
      Devices  : 1
          Device [#0]:
          Type       : gpu
          Version    : 1.3
          Name       : Intel(R) UHD Graphics 630 [0x3e92]
          Vendor     : Intel(R) Corporation
          Driver     : 1.3.22577
  Platform [#4]:
      Version  : 1.2
      Name     : SYCL host platform
      Vendor   : 
      Devices  : 1
          Device [#0]:
          Type       : host
          Version    : 1.2
          Name       : SYCL host device
          Vendor     : 
          Driver     : 1.2
  default_selector()      : gpu, Intel(R) Level-Zero, Intel(R) UHD Graphics 630 [0x3e92] 1.3 [1.3.22577]
  host_selector()         : host, SYCL host platform, SYCL host device 1.2 [1.2]
  accelerator_selector()  : No device of requested type available. -1 (CL_DEVI...
  cpu_selector()          : cpu, Intel(R) OpenCL, Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz 3.0 [2021.13.11.0.23_160000]
  gpu_selector()          : gpu, Intel(R) Level-Zero, Intel(R) UHD Graphics 630 [0x3e92] 1.3 [1.3.22577]
  custom_selector(gpu)    : gpu, Intel(R) Level-Zero, Intel(R) UHD Graphics 630 [0x3e92] 1.3 [1.3.22577]
  custom_selector(cpu)    : cpu, Intel(R) OpenCL, Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz 3.0 [2021.13.11.0.23_160000]
  custom_selector(acc)    : No device of requested type available. -1 (CL_DEVI...

Pull request, that triggered failure: #5756

[asudarsa]

Enabled the test back as I was not able to reproduce it. intel/llvm-test-suite#1035

[bader]

@asudarsa, it failed again in pre-commit for #7905.
https://github.com/intel/llvm/actions/runs/3830797091/jobs/6519512553

[bader]

One more case: https://github.com/intel/llvm/actions/runs/5183919838/jobs/9342780841.

GitHub Actions logs expire in 90 days. I'm attaching logs here in case someone will need logs after expiration date.

[AlexeySachkov]

Reproduced once more in an unrelated change: https://github.com/intel/llvm/actions/runs/9860326725/job/27230313679?pr=14451

-- Testing: 2102 tests, 8 workers --
FAIL: SYCL :: Plugin/level_zero_batch_event_status.cpp (1613 of 2102)
******************** TEST 'SYCL :: Plugin/level_zero_batch_event_status.cpp' FAILED ********************
Exit Code: 1

Command Output (stdout):
--
# RUN: at line 4
/__w/llvm/llvm/toolchain/bin//clang++   -fsycl -fsycl-targets=spir64  /__w/llvm/llvm/llvm/sycl/test-e2e/Plugin/level_zero_batch_event_status.cpp -o /__w/llvm/llvm/build-e2e/Plugin/Output/level_zero_batch_event_status.cpp.tmp.out
# executed command: /__w/llvm/llvm/toolchain/bin//clang++ -fsycl -fsycl-targets=spir64 /__w/llvm/llvm/llvm/sycl/test-e2e/Plugin/level_zero_batch_event_status.cpp -o /__w/llvm/llvm/build-e2e/Plugin/Output/level_zero_batch_event_status.cpp.tmp.out
# note: command had no output on stdout or stderr
# RUN: at line 7
env SYCL_PI_LEVEL_ZERO_BATCH_SIZE=4 SYCL_PI_LEVEL_ZERO_DEVICE_SCOPE_EVENTS=2 SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=0 SYCL_PI_TRACE=2 UR_L0_DEBUG=1 env ONEAPI_DEVICE_SELECTOR=level_zero:gpu  /__w/llvm/llvm/build-e2e/Plugin/Output/level_zero_batch_event_status.cpp.tmp.out 2>&1 | /__w/llvm/llvm/toolchain/bin/FileCheck /__w/llvm/llvm/llvm/sycl/test-e2e/Plugin/level_zero_batch_event_status.cpp
# executed command: env SYCL_PI_LEVEL_ZERO_BATCH_SIZE=4 SYCL_PI_LEVEL_ZERO_DEVICE_SCOPE_EVENTS=2 SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=0 SYCL_PI_TRACE=2 UR_L0_DEBUG=1 env ONEAPI_DEVICE_SELECTOR=level_zero:gpu /__w/llvm/llvm/build-e2e/Plugin/Output/level_zero_batch_event_status.cpp.tmp.out
# note: command had no output on stdout or stderr
# executed command: /__w/llvm/llvm/toolchain/bin/FileCheck /__w/llvm/llvm/llvm/sycl/test-e2e/Plugin/level_zero_batch_event_status.cpp
# .---command stderr------------
# | /__w/llvm/llvm/llvm/sycl/test-e2e/Plugin/level_zero_batch_event_status.cpp:30:11: error: CHECK: expected string not found in input
# | // CHECK: ---> piEnqueueKernelLaunch
# |           ^
# | <stdin>:741:20: note: scanning from here
# | ---> piEventGetInfo(
# |                    ^
# | <stdin>:775:1: note: possible intended match here
# | ---> piQueueRelease(
# | ^
# | 
# | Input file: <stdin>
# | Check file: /__w/llvm/llvm/llvm/sycl/test-e2e/Plugin/level_zero_batch_event_status.cpp
# | 
# | -dump-input=help explains the following input dump.
# | 
# | Input was:
# | <<<<<<
# |             .
# |             .
# |             .
# |           7[36](https://github.com/intel/llvm/actions/runs/9860326725/job/27230313679?pr=14451#step:22:37): UR <--- CleanupCompletedEvent(Event, QueueLocked, true)(UR_RESULT_SUCCESS) 
# |           7[37](https://github.com/intel/llvm/actions/runs/9860326725/job/27230313679?pr=14451#step:22:38): UR ---> urEventReleaseInternal(Event) 
# |           738: UR <--- urEventReleaseInternal(Event)(UR_RESULT_SUCCESS) 
# |           7[39](https://github.com/intel/llvm/actions/runs/9860326725/job/27230313679?pr=14451#step:22:40): ) ---> pi_result : PI_SUCCESS 
# |           7[40](https://github.com/intel/llvm/actions/runs/9860326725/job/27230313679?pr=14451#step:22:41):  
# |           741: ---> piEventGetInfo( 
# | check:30'0                        X~ error: no match found
# |           7[42](https://github.com/intel/llvm/actions/runs/9860326725/job/27230313679?pr=14451#step:22:43):  pi_event : 0x308c540 
# | check:30'0     ~~~~~~~~~~~~~~~~~~~~~~
# |           7[43](https://github.com/intel/llvm/actions/runs/9860326725/job/27230313679?pr=14451#step:22:44):  <unknown> : 4563 
# | check:30'0     ~~~~~~~~~~~~~~~~~~
# |           744:  <unknown> : 4 
# | check:30'0     ~~~~~~~~~~~~~~~
# |           745:  <unknown> : 0x7ffdb44c9844 
# | check:30'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |           746:  <nullptr> 
# | check:30'0     ~~~~~~~~~~~
# |             .
# |             .
# |             .
# |           770: UR ---> urQueueReleaseInternal(Queue) 
# | check:30'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |           771: UR <--- urQueueReleaseInternal(Queue)(UR_RESULT_SUCCESS) 
# | check:30'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |           772: UR <--- urEventReleaseInternal(Event)(UR_RESULT_SUCCESS) 
# | check:30'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |           773: ) ---> pi_result : PI_SUCCESS 
# | check:30'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |           774:  
# | check:30'0     ~
# |           775: ---> piQueueRelease( 
# | check:30'0     ~~~~~~~~~~~~~~~~~~~~~
# | check:30'1     ?                     possible intended match
# |           776:  <unknown> : 0x2f12810 
# | check:30'0     ~~~~~~~~~~~~~~~~~~~~~~~
# |           777: UR ---> Queue->synchronize() 
# | check:30'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |           778: UR <--- Queue->synchronize()(UR_RESULT_SUCCESS) 
# | check:30'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |           779: ZE ---> zeFenceDestroy(it->second.ZeFence) 
# | check:30'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |           780: UR ---> urQueueReleaseInternal(Queue) 
# | check:30'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |             .
# |             .
# |             .
# | >>>>>>
# `-----------------------------
# error: command failed with exit status: 1

--