Make fail in CPU/GPU mode - CUDA 10.1 - THUNDERconfig.h not found?

[tallgeese84]

Hello,

I tried compiling with CUDA 10.1/CMAKE3.5/gcc5.4.0 and when executing make -j 10
I got this error: fatal error: THUNDERCONFIG.h: No such file or directory

I can compile successfully with CPU only but not with GPU.

I am trying not to downgrade to CUDA 9 and hope that there will be an easy fix to this.

Many thanks in advance!

[Zarrathustra]

Hi,

There are some issues in compiling with CUDA 10.1.

There is a table of THUNDER compilation status under various environments. You may find it useful.

best wishes.

Mingxu

[Zarrathustra]

Moreover, there are bugs in version 3.5 of CMake. Please upgrade or downgrade CMake.

[tallgeese84]

Hello Mingxu,

Thank you for the info!
I tried cmake 3.3 but still encounter THUNDERconfig.h not found. This is on a Ubuntu 16.04 system.
Tried either openmpi or mpich, gcc 4.8.5, 5.4, 7.1, still passing the THUNDERconfig.h not found.

Any suggestions that I should try? Thank you in advance.

Best,
CJ

[Zarrathustra]

Hi,

What version are you compiling? I believe that we have already fixed this issue.

Mingxu

[tallgeese84]

Hi Mingxu,

I am compiling the Thunder 1.4.14. My Cuda is 10.1 and NCCL is 2.4.7

Best,
CJ

[tallgeese84]

I thought this might be helpful
from cmake ..

-- Welcome to THUNDER 1.4.14!
-- Git commit version:76d6a11f4fb45865e075c0e18213e131def703a4
-- Build THUNDER in single-float precision.
-- Try to build THUNDER in CPU & GPU version.
-- CMAKE_BUILD_TYPE : release, compile THUNDER with -O2 flag.
-- Build THUNDER using AVX256.
-- CMAKE_C_FLAGS : -Wall -Wno-uninitialized -Wno-deprecated-declarations -Wno-sign-compare -pthread -fopenmp -O2 -mavx
-- CMAKE_CXX_FLAGS : -Wall -Wno-uninitialized -Wno-deprecated-declarations -Wno-sign-compare -pthread -fopenmp -O2 -mavx
-- MPI_COMPILER : /home/WS1/mpich-3.2-build/bin/mpicxx
-- MPI_INCLUDE_PATH : /home/WS1/mpich-3.2-build/include
-- MPI_LIBRARIES : /home/WS1/mpich-3.2-build/lib/libmpicxx.so;/home/WS1/mpich-3.2-build/lib/libmpi.so
-- MPI_CXX_INCLUDE_PATH : /home/WS1/mpich-3.2-build/include
-- MPI_CXX_COMPILER : /home/WS1/mpich-3.2-build/bin/mpicxx
-- MPI_CXX_LIBRARIES : /home/WS1/mpich-3.2-build/lib/libmpicxx.so;/home/WS1/mpich-3.2-build/lib/libmpi.so
-- Getting ready for building FFTW-3
-- FFTW_FLAGS : --disable-doc;--enable-threads;--enable-float;--enable-avx;--prefix=/home/WS1/THUNDER/build/external/fftw
-- FFTW_LIBRARIES : /home/WS1/THUNDER/build/external/fftw/lib/libfftw3f.a;/home/WS1/THUNDER/build/external/fftw/lib/libfftw3f_threads.a
-- Getting ready for building GSL
-- GSL_LIBRARIES : /home/WS1/THUNDER/build/external/gsl/lib/libgsl.a;/home/WS1/THUNDER/build/external/gsl/lib/libgslcblas.a
-- Getting ready for building jsoncpp
-- CUDA_VERSION : 10.1
-- CUDA_TOOLKIT_ROOT_DIR : /usr/local/cuda-10.1
-- CUDA_INCLUDE_DIRS : /usr/local/cuda-10.1/include
-- CUDA_LIBRARIES : /usr/local/cuda-10.1/lib64/libcudart_static.a-lpthread/usr/lib/x86_64-linux-gnu/librt.so/usr/lib/x86_64-linux-gnu/libdl.so
-- CUDA_CUBLAS_LIBRARIES : /usr/lib/x86_64-linux-gnu/libcublas.so
-- CUDA_curand_LIBRARY : /usr/local/cuda-10.1/lib64/libcurand.so
-- CUDA_CUFFT_LIBRARIES : /usr/local/cuda-10.1/lib64/libcufft.so
-- CUDA_NVCC_FLAGS : -gencode=arch=compute_30,code=sm_30-gencode=arch=compute_35,code=sm_35-gencode=arch=compute_50,code=sm_50-gencode=arch=compute_52,code=sm_52-gencode=arch=compute_53,code=sm_53-gencode=arch=compute_60,code=sm_60-std=c++11-Wno-deprecated-gpu-targets
-- NCCL_INCLUDES : /usr/include
-- NCCL_LIBRARIES : /usr/lib/x86_64-linux-gnu/libnccl_static.a
-- Externel lib: /usr/local/cuda-10.1/lib64/libcudart_static.a-lpthread/usr/lib/x86_64-linux-gnu/librt.so/usr/lib/x86_64-linux-gnu/libdl.so/usr/lib/x86_64-linux-gnu/libcublas.so/usr/local/cuda-10.1/lib64/libcufft.so/usr/local/cuda-10.1/lib64/libcurand.so/usr/lib/x86_64-linux-gnu/libnccl_static.a
-- on
-- Configuring done
-- Generating done
-- Build files have been written to: /home/WS1/THUNDER/build

[Zarrathustra]

Sorry, it will take some time to repeat your environment and checkout out the problem. It may save your time to try to compile THUNDER in the environment that we used in cryoEM facility in Tsinghua, which is

1. gcc 4.8.5 (CentOS7)
2. mpich3.3
3. CUDA 9.1 + corresponding NCCL
4. cmake 3.12

In my opinion, the most likely issue is CMake version and CUDA. If possible, may you try CMake 3.12 and CUDA 9.1?

Best wishes.

Mingxu

[thunder_user@gpu00 build]$ cmake ..
-- The C compiler identification is GNU 4.8.5
-- The CXX compiler identification is GNU 4.8.5
-- Check for working C compiler: /Share/THUDATA/Softwares/mpich3.3/bin/mpicc
-- Check for working C compiler: /Share/THUDATA/Softwares/mpich3.3/bin/mpicc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /Share/THUDATA/Softwares/mpich3.3/bin/mpicxx
-- Check for working CXX compiler: /Share/THUDATA/Softwares/mpich3.3/bin/mpicxx -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Welcome to THUNDER 1.4.14!
-- Git commit version:76d6a11f4fb45865e075c0e18213e131def703a4
-- Build THUNDER in single-float precision.
-- Try to build THUNDER in CPU & GPU version.
-- CMAKE_BUILD_TYPE : release, compile THUNDER with -O2 flag.
-- Build THUNDER using AVX256.
-- CMAKE_C_FLAGS : -Wall -Wno-uninitialized -Wno-deprecated-declarations -Wno-sign-compare -pthread -fopenmp -O2 -mavx
-- CMAKE_CXX_FLAGS : -Wall -Wno-uninitialized -Wno-deprecated-declarations -Wno-sign-compare -pthread -fopenmp -O2 -mavx
CMake Warning (dev) at CMakeLists.txt:156 (find_package):
Policy CMP0074 is not set: find_package uses PackageName_ROOT variables.
Run "cmake --help-policy CMP0074" for policy details. Use the cmake_policy
command to set the policy and suppress this warning.

Environment variable MPI_ROOT is set to:

/Share/THUDATA/Softwares/mpich3.3

For compatibility, CMake is ignoring the variable.
This warning is for project developers. Use -Wno-dev to suppress it.

-- Found MPI_C: /Share/THUDATA/Softwares/mpich3.3/bin/mpicc (found version "3.1")
-- Found MPI_CXX: /Share/THUDATA/Softwares/mpich3.3/bin/mpicxx (found version "3.1")
-- Found MPI: TRUE (found version "3.1")
-- MPI_COMPILER : /Share/THUDATA/Softwares/mpich3.3/bin/mpicxx
-- MPI_INCLUDE_PATH :
-- MPI_LIBRARIES :
-- MPI_CXX_INCLUDE_PATH :
-- MPI_CXX_COMPILER : /Share/THUDATA/Softwares/mpich3.3/bin/mpicxx
-- MPI_CXX_LIBRARIES :
-- Getting ready for building FFTW-3
-- FFTW_FLAGS : --disable-doc;--enable-threads;--enable-float;--enable-avx;--prefix=/Share/THUDATA/UserHome/thunder_user/mingxu/Compilation_Test/THUNDER/build/external/fftw
-- FFTW_LIBRARIES : /Share/THUDATA/UserHome/thunder_user/mingxu/Compilation_Test/THUNDER/build/external/fftw/lib/libfftw3f.a;/Share/THUDATA/UserHome/thunder_user/mingxu/Compilation_Test/THUNDER/build/external/fftw/lib/libfftw3f_threads.a
-- Getting ready for building GSL
-- GSL_LIBRARIES : /Share/THUDATA/UserHome/thunder_user/mingxu/Compilation_Test/THUNDER/build/external/gsl/lib/libgsl.a;/Share/THUDATA/UserHome/thunder_user/mingxu/Compilation_Test/THUNDER/build/external/gsl/lib/libgslcblas.a
-- Getting ready for building jsoncpp
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found CUDA: /Share/THUDATA/Softwares/cuda-9.1.85 (found suitable version "9.1", minimum required is "8.0")
-- Found NCCL: /Share/THUDATA/Softwares/cuda-9.1.85/include
-- CUDA_VERSION : 9.1
-- CUDA_TOOLKIT_ROOT_DIR : /Share/THUDATA/Softwares/cuda-9.1.85
-- CUDA_INCLUDE_DIRS : /Share/THUDATA/Softwares/cuda-9.1.85/include
-- CUDA_LIBRARIES : /Share/THUDATA/Softwares/cuda-9.1.85/lib64/libcudart_static.a-lpthreaddl/usr/lib64/librt.so
-- CUDA_CUBLAS_LIBRARIES : /Share/THUDATA/Softwares/cuda-9.1.85/lib64/libcublas.so/Share/THUDATA/Softwares/cuda-9.1.85/lib64/libcublas_device.a
-- CUDA_curand_LIBRARY : /Share/THUDATA/Softwares/cuda-9.1.85/lib64/libcurand.so
-- CUDA_CUFFT_LIBRARIES : /Share/THUDATA/Softwares/cuda-9.1.85/lib64/libcufft.so
-- CUDA_NVCC_FLAGS : -gencode=arch=compute_30,code=sm_30-gencode=arch=compute_35,code=sm_35-gencode=arch=compute_50,code=sm_50-gencode=arch=compute_52,code=sm_52-gencode=arch=compute_53,code=sm_53-gencode=arch=compute_60,code=sm_60-gencode=arch=compute_70,code=sm_70-std=c++11-Wno-deprecated-gpu-targets
-- NCCL_INCLUDES : /Share/THUDATA/Softwares/cuda-9.1.85/include
-- NCCL_LIBRARIES : /Share/THUDATA/Softwares/cuda-9.1.85/lib64/libnccl_static.a
-- Externel lib: /Share/THUDATA/Softwares/cuda-9.1.85/lib64/libcudart_static.a-lpthreaddl/usr/lib64/librt.so/Share/THUDATA/Softwares/cuda-9.1.85/lib64/libcublas.so/Share/THUDATA/Softwares/cuda-9.1.85/lib64/libcublas_device.a/Share/THUDATA/Softwares/cuda-9.1.85/lib64/libcufft.so/Share/THUDATA/Softwares/cuda-9.1.85/lib64/libcurand.so/Share/THUDATA/Softwares/cuda-9.1.85/lib64/libnccl_static.a
-- on
-- Configuring done
-- Generating done
-- Build files have been written to: /Share/THUDATA/UserHome/thunder_user/mingxu/Compilation_Test/THUNDER/build

[tallgeese84]

No worries! Thanks for the suggestions! I will try the newer cmake first before testing cuda-9.

Best,
CJ

[Zarrathustra]

Hi,

We have found the cause of your issue. We have fixed it in the source code.

Renew your source code (git pull or redo git clone), please. Your compilation issue should be solved then.

Best wishes,

Mingxu

[tallgeese84]

Hi Mingxu,

Thanks! I think your new code fixed the problem but a new problem came up right at the end
[ 5%] Built target GSL
[ 11%] Built target FFTW
[ 13%] Built target jsoncpp
[ 15%] Built target cuthuem
[ 15%] Linking CXX static library ../../lib/libcuthuemcore.a
[ 38%] Built target core_CPU
[ 51%] Built target cuthuemcore
[ 53%] Built target thunder_postprocess
[ 54%] Built target thunder_lowpass
[ 55%] Built target thunder_bfactor
[ 56%] Built target thunder_minus
[ 57%] Built target thunder_genmask
[ 58%] Built target thunder_alignZ
[ 60%] Built target thunder_mask
[ 64%] Built target thunder_genmask_shell
[ 64%] Built target thunder_average
[ 66%] Built target thunder_cpu
[ 67%] Built target thunder_project
[ 68%] Linking CXX executable ../../cutest/ReconstructorAccTest
[ 68%] Linking CXX executable ../../cutest/SimpleCMakeTest
[ 69%] Linking CXX executable ../../cutest/CastTest
[ 71%] Built target thunder_view
[ 73%] Built target thunder_reconstruct
[ 74%] Built target thunder_resize
[ 75%] Built target SimpleCMakeTest
[ 75%] Built target CastTest
[ 98%] Built target core_GPU
gpu/testsrc/CMakeFiles/ReconstructorAccTest.dir/build.make:112: recipe for target 'cutest/ReconstructorAccTest' failed
CMakeFiles/Makefile2:1061: recipe for target 'gpu/testsrc/CMakeFiles/ReconstructorAccTest.dir/all' failed
Scanning dependencies of target thunder_gpu
[ 98%] Building CXX object CMakeFiles/thunder_gpu.dir/appsrc/thunder.cpp.o
[ 99%] Linking CXX executable thunder_gpu
CMakeFiles/thunder_gpu.dir/build.make:115: recipe for target 'thunder_gpu' failed
CMakeFiles/Makefile2:450: recipe for target 'CMakeFiles/thunder_gpu.dir/all' failed
Makefile:127: recipe for target 'all' failed

Have you encounter this make issue before?

Best,
CJ

[Zarrathustra]

Hi,

Sorry, we have not encountered such make issue before. What environment are you currently compiling under?

From your make output, it seems that something wrong happened during compiling UnitTest code for GPU, which is not useful for you. You can disable the compilation of this part and have another try.

Modify file THUNDER/gpu/CMakeLists.txt by deleting ADD_SUBDIRECTORY(testsrc) (line 22), please.

Best wishes.

Mingxu

[tallgeese84]

Hi Mingxu,

The compilation did not work but atleast we got the CPU version to work on our clusters so that should be fine for now. The current compiling environment on our own GPU workstation is ubuntu 16.04, cuda 10.1, cmake 3.3, mpich-3.3.1, gcc 4.8.5. I might try to reinstall the cuda and see if that solves the compilation issue using GPU.

Thanks again for your help!

Best,
CJ