OpenMP offloaded code crashes at compilation/runtime depending on optimization flags

[JakobSchaeffeler]

I'm trying to compile the stencil3d-omp benchmark of HeCBench: https://github.com/zjin-lcf/HeCBench/blob/master/src/stencil3d-omp/main.cpp

I'm using LLVM version 19.1.3 and I'm offloading to a MI100 AMD GPU.

If I compile the code with -O3 everything works and the results match with the ones from SYCL and HIP:
make CC=clang++ CFLAGS="-fopenmp -fopenmp-targets=amdgcn-amd-amdhsa -Xopenmp-target=amdgcn-amd-amdhsa -march=gfx908"

If I compile the code without any optimization flags the compilation is successful, however at runtime i get the following error:
AMDGPU fatal error 1: Memory access fault by GPU 8 (agent 0x55f222c19fb0) at virtual address 0x7f8c9c06a000. Reasons: Page not present or supervisor privilege Aborted (core dumped)
Lastly, if I compile the code with O0, O1 or O2 I get a Segfault at compilation: O0_compilation_output.txt

The only difference I was able to find between the O0 and O3 version is that the O0 version launches the OpenMP kernels in generic mode and the O3 in generic-SPMD mode. Could this be the reason for the crash?

[llvmbot]

@llvm/issue-subscribers-offload

Author: None (JakobSchaeffeler)

I'm trying to compile the stencil3d-omp benchmark of HeCBench: [https://github.com/zjin-lcf/HeCBench/blob/master/src/stencil3d-omp/main.cpp](https://github.com/zjin-lcf/HeCBench/blob/master/src/stencil3d-omp/main.cpp)

I'm using LLVM version 19.1.3 and I'm offloading to a MI100 AMD GPU.

If I compile the code with -O3 everything works and the results match with the ones from SYCL and HIP:
make CC=clang++ CFLAGS="-fopenmp -fopenmp-targets=amdgcn-amd-amdhsa -Xopenmp-target=amdgcn-amd-amdhsa -march=gfx908"

If I compile the code without any optimization flags the compilation is successful, however at runtime i get the following error:
AMDGPU fatal error 1: Memory access fault by GPU 8 (agent 0x55f222c19fb0) at virtual address 0x7f8c9c06a000. Reasons: Page not present or supervisor privilege Aborted (core dumped)
Lastly, if I compile the code with O0, O1 or O2 I get a Segfault at compilation: O0_compilation_output.txt

The only difference I was able to find between the O0 and O3 version is that the O0 version launches the OpenMP kernels in generic mode and the O3 in generic-SPMD mode. Could this be the reason for the crash?

[jhuber6]

Offloading failing at O0 is quite common, as with no optimizations you're more liable to run out of stack space or some other resource. The compiler crashing is definitely more concerning, seems to be crashing in AMDGPUResourceUsageAnalysis. Could you compile with -save-temps and provide the IR? Something that reproduced via https://godbolt.org/z/hj3KEjPfd would be ideal, but just having the GPU IR would help.

The only difference I was able to find between the O0 and O3 version is that the O0 version launches the OpenMP kernels in generic mode and the O3 in generic-SPMD mode. Could this be the reason for the crash?

No, that's just an optimization that's run at O1, it should work without the transform but it will be a lot slower.

[JakobSchaeffeler]

Thanks for the quick reply!

I attached the .bc files I got with -save-temps and the .ll files i created from the bc files with llvm-dis. I also tried to put the LLVM IR files into godbolt, but for me they did not fail to compile there.

main.amdgcn-amd-amdhsa.gfx908.postlink.bc.txt
main.amdgcn-amd-amdhsa.gfx908.postlink.ll.txt
main.amdgcn-amd-amdhsa.gfx908.postopt.bc.txt
main.amdgcn-amd-amdhsa.gfx908.postopt.ll.txt

[jhuber6]

I didn't get a crash with the provided IR using the most up-to-date clang. Compiling your source, I saw it crash at low optimization levels but only when the dimension got sufficiently big, so I'm going to assume that the GPU is just running out of resources at some point.