CUDA_ERROR_ILLEGAL_ADDRESS in wait()

[mmeterel]

I am working to enable cublasSgemmBatched (and other precisions) for CUDA backend in oneMKL open source interfaces, and I am seeing the below error when I call queue.wait() or event.wait().

PI CUDA ERROR:
Value: 700
Name: CUDA_ERROR_ILLEGAL_ADDRESS
Description: an illegal memory access was encountered
Function: wait
Source Location: /nfs/pdx/disks/mkl_project/mmeterel/tools/dpcpp/llvm/sycl/plugins/cuda/pi_cuda.cpp:374

Error points to cuEventSynchronize() in wait().

pi_result pi_event::wait() {
pi_result retErr;
try {
retErr = PI_CHECK_ERROR(cuEventSynchronize(evEnd));
isCompleted_ = true;
} catch (pi_result error) {
retErr = error;
}

return retErr;
}

[bader]

@mmeterel, could you clarify what version of the compiler/runtime do you use?

[mmeterel]

@bader
Compiler: I tried local builds from 20200608 and 20200527. I see the issue in both.

Runtime: How can I check the runtime version?

[bader]

Compiler: I tried local builds from 20200608 and 20200527. I see the issue in both.

Runtime: How can I check the runtime version?

If you followed the instructions from Get Started Guide, the version is the same. You local build should have both components: compiler and runtime.

Thanks for taking your time to report the issue.

[mmeterel]

Hi, @bader
Is the information I provided (date) sufficient or should i try to find the exact version number for runtime? Just making sure I understand the question correctly.

[bader]

I was trying to understand how old is the version and I think I get the answer - it's quite recent and most likely it's not fixed yet. Unfortunately, I have no clue what might cause this issue, so I'll let @Ruyk or @bjoernknafla to comment.

[Ruyk]

" CUDA_ERROR_ILLEGAL_ADDRESS" usually occurs when accessing out of bonds from a CUDA kernel.
CUDA errors are asynchronous, so they are reported on the next synchronization point, in this case the wait.
Seems to me this is an issue with the parameters of your oneMKL call and not a DPC++ problem - there is little in CUDA backend that could case that problem.
Try using cuda debugger or cuda-memcheck, they usually help tracking down this kind of error.

[mmeterel]

After some debug, I noticed that, the error happens when the memory for the input matrices is allocated using malloc_shared(). The error is gone when I use usm_allocator<T, usm::alloc::shared, 64>.

Not sure if this is the expected behavior or hints a potential problem with malloc_shared().

Feel free to close the issue.

[bader]

According to this document - https://github.com/intel/llvm/tree/sycl/sycl/doc/extensions#extensions, USM extension is not supported by CUDA back-end, so I guess issues like this are expected, but probably more clear error message would be preferrable.
There is at least one open PR improving USM support for CUDA back-end (#1577), but I'm not sure if it addresses this particular issue.

Tagging @jbrodman and @fwyzard.

Should we keep this tracker open or create a new one covering all the issues with enabling USM feature for CUDA back-end (i.e. "Enable USM for CUDA back-end" feature request")?

[Ruyk]

I think a "feature-request" would be good, in case someone from the community wants to take this up at some point.

[bader]

@mmeterel, could you check if this issue is still reproducible the latest build, please?
#2557 enabled some USM functionality for CUDA back-end.

[hdelan]

@mmeterel I wasn't able to reproduce this error with a generic use of cublasSgemmBatched.

Could you please provide a minimal case that gave this error? Thank in advance.

[AerialMantis]

As we're not able to reproduce this issues described in here we believe it's like the issue has been solved by a prior change, so we would like to close this ticket. However, if the issue still appears please comment and we can re-open the ticket.