[cuBLAS][DPCPP] Build issue with cuBLAS backend and DPCPP

[abagusetty]

Summary

Build fails with CUBLAS backend and DPC++ with the following errors. However the build is just fine with CURAND backend. Any pointers how to navigate this.

cuda_fp16.hpp:690:1: error: unknown type name 'CUDA_FP16_DECL'

Debug info from CMake output: (cmake/FindcuBLAS.cmake)

-- Found CUDA: /soft/compilers/cudatoolkit/cuda-12.0.0 (found suitable version "12.0", minimum required is "10.0") 
-- Found cuBLAS: /soft/compilers/cudatoolkit/cuda-12.0.0/include  
-- 2. CUDA_TOOLKIT_INCLUDE : /soft/compilers/cudatoolkit/cuda-12.0.0/include
-- 2. CUDA_cublas_LIBRARY  : /soft/compilers/cudatoolkit/cuda-12.0.0/lib64/libcublas.so
-- 2. CUDA_LIBRARIES  : /soft/compilers/cudatoolkit/cuda-12.0.0/lib64/libcudart_static.a;Threads::Threads;dl;/usr/lib64/librt.so
-- 2. CUDA_CUDART_LIBRARY  : /soft/compilers/cudatoolkit/cuda-12.0.0/lib64/libcudart.so
-- 2. CUDA_CUDA_LIBRARY  : /soft/compilers/cudatoolkit/cuda-12.0.0/lib64/stubs/libcuda.so

Version

oneMKL: d217915
llvm/dpcpp: b2d6fdfd63fe8dd7be98e113cde9bc7c3d9a21d8
CUDA: 12.0

Build instructions

CUDA_LIB_PATH=/soft/compilers/cudatoolkit/cuda-12.0.0/lib64/stubs cmake .. -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DENABLE_CUBLAS_BACKEND=True -DENABLE_CURAND_BACKEND=True -DENABLE_ROCBLAS_BACKEND=False -DENABLE_MKLCPU_BACKEND=False -DENABLE_MKLGPU_BACKEND=False -DBUILD_FUNCTIONAL_TESTS=False -DBUILD_EXAMPLES=False

Error

[ 26%] Building CXX object bin/blas/backends/cublas/CMakeFiles/onemkl_blas_cublas_obj.dir/cublas_level1.cpp.o
cd /lus/eagle/projects/UINTAH_aesp/abagusetty/oneMKL/build/bin/blas/backends/cublas && /lus/eagle/projects/UINTAH_aesp/abagusetty/llvm_sycl/build_PrngEnvGnu_cuda1200_12212022/install/bin/clang++ -DCUDA_NO_HALF -I/lus/eagle/projects/UINTAH_aesp/abagusetty/oneMKL/include -I/lus/eagle/projects/UINTAH_aesp/abagusetty/oneMKL/src/include -I/lus/eagle/projects/UINTAH_aesp/abagusetty/oneMKL/src -isystem /lus/eagle/projects/UINTAH_aesp/abagusetty/llvm_sycl/build_PrngEnvGnu_cuda1200_12212022/install/include/sycl -isystem /soft/compilers/cudatoolkit/cuda-12.0.0/include -DSYCL2020_DISABLE_DEPRECATION_WARNINGS -O3 -DNDEBUG -fPIC -fsycl -fsycl-targets=nvptx64-nvidia-cuda -fsycl-unnamed-lambda -MD -MT bin/blas/backends/cublas/CMakeFiles/onemkl_blas_cublas_obj.dir/cublas_level1.cpp.o -MF CMakeFiles/onemkl_blas_cublas_obj.dir/cublas_level1.cpp.o.d -o CMakeFiles/onemkl_blas_cublas_obj.dir/cublas_level1.cpp.o -c /lus/eagle/projects/UINTAH_aesp/abagusetty/oneMKL/src/blas/backends/cublas/cublas_level1.cpp
clang-16: warning: CUDA version is newer than the latest partially supported version 11.8 [-Wunknown-cuda-version]
In file included from /lus/eagle/projects/UINTAH_aesp/abagusetty/oneMKL/src/blas/backends/cublas/cublas_level1.cpp:20:
In file included from /lus/eagle/projects/UINTAH_aesp/abagusetty/oneMKL/src/blas/backends/cublas/cublas_task.hpp:13:
In file included from /lus/eagle/projects/UINTAH_aesp/abagusetty/oneMKL/src/blas/backends/cublas/cublas_scope_handle.hpp:27:
/lus/eagle/projects/UINTAH_aesp/abagusetty/llvm_sycl/build_PrngEnvGnu_cuda1200_12212022/install/bin/../include/sycl/backend/cuda.hpp:17:2: warning: sycl/backend/cuda.hpp is deprecated and not required anymore [-W#warnings]
#warning sycl/backend/cuda.hpp is deprecated and not required anymore
 ^
1 warning generated.
In file included from /lus/eagle/projects/UINTAH_aesp/abagusetty/oneMKL/src/blas/backends/cublas/cublas_level1.cpp:19:
In file included from /lus/eagle/projects/UINTAH_aesp/abagusetty/oneMKL/src/blas/backends/cublas/cublas_helper.hpp:31:
In file included from /soft/compilers/cudatoolkit/cuda-12.0.0/include/cublas_v2.h:69:
In file included from /soft/compilers/cudatoolkit/cuda-12.0.0/include/cublas_api.h:77:
In file included from /soft/compilers/cudatoolkit/cuda-12.0.0/include/cuda_fp16.h:4006:
/soft/compilers/cudatoolkit/cuda-12.0.0/include/cuda_fp16.hpp:690:1: error: unknown type name '__CUDA_FP16_DECL__'
__CUDA_FP16_DECL__ __half2 __internal_device_float2_to_half2_rn(const float a, const float b) {
^
/soft/compilers/cudatoolkit/cuda-12.0.0/include/cuda_fp16.hpp:690:27: error: expected ';' after top level declarator
__CUDA_FP16_DECL__ __half2 __internal_device_float2_to_half2_rn(const float a, const float b) {
                          ^

[mmeterel]

@abagusetty Thanks for reporting this issue.
Do you know if the same problem occurs with CUDA versions 10.2 or 11.x?

[abagusetty]

@mmeterel Only with 11.x.

[mmeterel]

Do you mean, the error don't appear with 10.2? And appears with 11.x (and 12.0 as shown in your logs)?

[abagusetty]

Yes that is correct. Any version > 11.x including 12.0 only shows this issue. Had no problems with 10.2.89. This is only with cublas backend. No issues with rng domain(i.e, disabled cublas and enabled curand).

I'm was sure if the above issue is related to simple cmake path or more involved. So tried to output some of the cmake path variables used by cublas cmakelists.

[mmeterel]

Yes that is correct. Any version > 11.x including 12.0 only shows this issue. Had no problems with 10.2.89. This is only with cublas backend. No issues with rng domain(i.e, disabled cublas and enabled curand).

I'm was sure if the above issue is related to simple cmake path or more involved. So tried to output some of the cmake path variables used by cublas cmakelists.

Thanks for the information. Let me try to reproduce in on my side.

[abagusetty]

Hi @mmeterel Just curious if you had a chance to reproduce the issue. Also was thinking if this has anything to do with how DPC++ is built (i.e., the bf16 support) etc. I was just using the standard CUDA-plugin instructions for DPC++

[abagusetty]

I was able to fix the above build issue when I've played with CUDA_LIB_PATH stubs location. Not sure if that makes sense.

CUDA_LIB_PATH=/soft/compilers/cudatoolkit/cuda-12.0.0/lib64/stubs to CUDA_LIB_PATH=/soft/compilers/cudatoolkit/cuda-12.0.0/targets/x86_64-linux/lib/stubs

[mmeterel]

@abagusetty Terribly sorry for not getting back to you earlier. :(

I tried the below combination and did not see the build error you mentioned.

oneMKL interfaces commit 45c43ed
Machine: A100
CUDA version: cuda_12.0.r12.0

One difference I have is, I am using an old llvm commit (having difficulty in detecting GPU device with newer version). I will try a newer version when I figure out the issue.

Also note that I did not specify 'CUDA_LIB_PATH'.

I noticed you have this '-DENABLE_CUBLAS_BACKEND=True -DENABLE_CURAND_BACKEND=True'. Are you purposely trying to build both backends? AFAIK, the repo do not support that currently - although your problem is probably not due to this.

Finally, I tried to compare my Cmake output with yours, but could not locate mine. How can I find that info?

[abagusetty]

@mmeterel Thanks for taking time on this. I was trying to build both CUBLAS and CURAND since the TARGET_DOMAINS have the option to put a comma separated values.

Was able to successfully build ROCBLAS and ROCRAND on the other hand without any issues. Regarding the cmake output, I just outputted the values of these variables (listed below) manually for debugging purposes. (i.e., edited FindCublas.cmake file)

-- 2. CUDA_TOOLKIT_INCLUDE : /soft/compilers/cudatoolkit/cuda-12.0.0/include
-- 2. CUDA_cublas_LIBRARY  : /soft/compilers/cudatoolkit/cuda-12.0.0/lib64/libcublas.so
-- 2. CUDA_LIBRARIES  : /soft/compilers/cudatoolkit/cuda-12.0.0/lib64/libcudart_static.a;Threads::Threads;dl;/usr/lib64/librt.so
-- 2. CUDA_CUDART_LIBRARY  : /soft/compilers/cudatoolkit/cuda-12.0.0/lib64/libcudart.so
-- 2. CUDA_CUDA_LIBRARY  : /soft/compilers/cudatoolkit/cuda-12.0.0/lib64/stubs/libcuda.so

[mmeterel]

Thanks for cmake tip. Here is the info I see:
-- CUDA_TOOLKIT_INCLUDE: /usr/local/cuda/include
-- CUDA_cublas_LIBRARY: /usr/local/cuda/lib64/libcublas.so
-- CUDA_LIBRARIES: /usr/local/cuda/lib64/libcudart_static.a;dl;/usr/lib/x86_64-linux-gnu/librt.a
-- CUDA_CUDART_LIBRARY: /usr/local/cuda/lib64/libcudart.so
-- CUDA_CUDA_LIBRARY: /usr/lib/x86_64-linux-gnu/libcuda.so

That does not help me much with your error frankly. Do you see anything?

[abagusetty]

Unfortunately, the standard installation paths for CUDA mightn't not provide the right setup. Since all the issues that were reported in this thread and others were all non-standard CUDA installation paths.

The above hack was on ALCF Polaris machine but I can try on NERSC Perlmutter to see if the CUDA_LIB_PATH setup is really the case of this build issue. If that is the case, a simple note in the documentation is all one might need. I can get back to you on this. Thanks again.

[mmeterel]

Thanks for following up. Yeah, I am curious what is the problem here :)

Also, I just noticed you are not building functional tests (-DBUILD_FUNCTIONAL_TESTS=False). In this case, you can build multiple backends. The issue will appear when you build the tests along with multiple backends.