smilei v5.0 problems with compiling on GPU on new HPC

[spadova-a]

Hello there,

I would like to use Smilei with GPU on Karolina cluster at IT4I in Ostrava and I am not sure how to compile it. So, I asked the administrator to help me with it, but he encoutered the following problem - GPU compilation for A100 fails with:

src/Diagnostic/DiagnosticScalar.cpp(802): error: expected a ";"
                                        maxval = fieldval;
                                        ^
src/Diagnostic/DiagnosticScalar.cpp(803): error #547: nonstandard form for taking the address of a member function
                              ATOMIC(write)
                                     ^
src/Diagnostic/DiagnosticScalar.cpp(804): error: expected a ";"
                              i_max=i;
                              ^
src/Diagnostic/DiagnosticScalar.cpp(805): error #547: nonstandard form for taking the address of a member function
                              ATOMIC(write)
                                     ^
src/Diagnostic/DiagnosticScalar.cpp(806): error: expected a ";"
                                        j_max=j;
                                        ^
src/Diagnostic/DiagnosticScalar.cpp(807): error #547: nonstandard form for taking the address of a member function
                              ATOMIC(write)
                                     ^
src/Diagnostic/DiagnosticScalar.cpp(808): error: expected a ";"
                                        k_max=k;

I will share this issue with the administrator, since I don't know any details of his procedure. Could you please help us find the problem? Note, that there were no problems with the CPU compilation.

[charlesprouveur]

Hello,
We will need a bit more information to help you, ie what make command did you use? Did you try to use a machine file? The documentation will be updated in the near future to better guide SMILEI's users through the compilation targeting GPU acceleration. In the meantime there has been a discussion for V100 on the element channel and you can see on this github an issue detailing the compilation process to target AMD GPU which should inspire you nonetheless.

Typically for a A100 you can look at the machine file:

smilei/scripts/compile_tools/machine/jean_zay_gpu_A100  

Your make command would look something like:

make -j 12 machine="jean_zay_gpu_A100" config="gpu_nvidia noopenmp verbose"

An example of a working environment we can recommend would be:

module purge
module load anaconda-py3/2020.11
module load nvidia-compilers/23.1
module load cuda/11.2
module load openmpi/4.1.1-cuda
module load hdf5/1.12.0-mpi-cuda
# For HDF5, note that module show can give you the right path
export HDF5_ROOT_DIR=/DIRECTORY_NAME/hdf5/1.12.0/pgi-20.4-HASH/

Regarding your specific error, it looks like you did not compile with nvc++, likely because no machine file for GPU was specified in the make command.
Edit: actually this is because you did not use a machine file: the compilation flags are missing "-DSMILEI_OPENACC_MODE"

[spadova-a]

Hi, sorry for the late answer, I will try to summarise what we tried.

Loaded modules:

Python/3.10.8-GCCcore-12.2.0 
HDF5/1.14.0-gompi-2022b 
NVHPC/23.7
CUDA/11.7

Created new machine file containing:

SMILEICXX.DEPS = nvcc
THRUSTCXX = nvcc

ACCELERATOR_GPU_FLAGS += -DSMILEI_OPENACC_MODE
ACCELERATOR_GPU_KERNEL_FLAGS += -DSMILEI_OPENACC_MODE

LDFLAGS += -ta=tesla:cc70 -std=c++14 -Mcudalib=curand -lcudart -lcurand -lacccuda -L${EBROOTCUDA}lib64/
CXXFLAGS +=  -D__GCC_ATOMIC_TEST_AND_SET_TRUEVAL=1
LDFLAGS = $LDFLAGS:$LD_LIBRARY_PATH
HDF5_ROOT_DIR = ${EBROOTHDF5}

and used the make command:
make -j 12 machine="karolina_IT4I" config="gpu_nvidia noopenmp verbose"

But still didn't have luck with the compilation, the error we are getting is:

src/Params/Params.h: In static member function ‘static constexpr int Params::getGPUClusterWidth(int)’:
src/Params/Params.h:421:5: error: body of ‘constexpr’ function ‘static constexpr int Params::getGPUClusterWidth(int)’ not a return-statement
  421 |     }
      |     ^

Do you have an idea, what could be the problem? wrong compilation flags, missing or incorrect module?

[charlesprouveur]

Hi,
Looking at the modules i do see a couple issues:

HDF5/1.14.0-gompi-2022b 

means your hdf5 module was not compiled with your nvhpc module / the nvidia compiler nvc++

Now i can already predict some issues with your nvhpc module that is recent:

NVHPC/23.7

it will require using '-gpu=cc70' as -ta=tesla:cc70 is deprecated after nvhpc 23.5, -Mcudalib=curand should also be removed as it is deprecated, we also know that we have an issue with the newest curand library therefore you will need a fix for the header file gpuRandom.h in src/tools/ ...
As for the cuda version i would recommend either 11.2 or 11.8.. We have currently an issue with CUDA >12.0

Finally for your specific error, could you print the command make is trying to execute (that you should be seeing thanks to the "verbose" configuration ) to be sure there is nothing else?

To sum things up: the quickest way for you to use smilei on gpu would be to:

1. use cuda <= 11.8 and nvhpc <= 23.1 (23.2 and 23.3 may work with no changes, i just have not tested this specific configuration before)
2. compile a hdf5 module with nvidia compiler

[spadova-a]

So, concerning the HDF5 - there is no module compiled with nvidia compiler and enable the paralell option at the cluster, this means I have to download and compile it myself, right?

About the CUDA - there is no CUDA 11.8 nor 11.2, will 11.3, 11.4 or 11.7 do?

Concerning the error, I am not sure, where I can find this. But these are the last lines and the error occurred, in fact, multiple times
make: *** [build/src/Diagnostic/DiagnosticFieldsAM.o] Error 1
make: *** [build/src/Diagnostic/DiagnosticFields2D.o] Error 1
make: *** [build/src/Diagnostic/DiagnosticTrack.o] Error 1
make: *** [build/src/Diagnostic/DiagnosticParticleList.o] Error 1
make: *** [build/src/Checkpoint/Checkpoint.o] Error 1
make: *** [build/src/Collisions/BinaryProcesses.o] Error 1

[beck-llr]

The way to go is normally to ask the administrator to make it available to you. It will benefit other potential users too.

[charlesprouveur]

Regarding hdf5, as beck-llr said, that should be the job of your support team/admins. (you would do something like this comment: https://forums.developer.nvidia.com/t/how-to-build-parallel-hdf5-with-nvhpc/181361/4 )

For the cuda version you mention it should not be a problem. You have to watch out for the nvhpc module though, do you have anything <= 23.1 ?

Finally, the errors you mentioned are due to make terminating because of the error you showed part of previously.
Can you show what

 make -j 1 machine="karolina_IT4I" config="gpu_nvidia noopenmp verbose"

returns you?

[spadova-a]

Hi,
ok, I will ask the support teams to compile the HDF5.

Yes, there is NVHPC/23.1, 22.7 and 22.2.

Still not sure, what exactly do you want me to show... this is everything that is written by the terminal:

Compiling src/Checkpoint/Checkpoint.cpp
mpicxx -Wno-reorder -D__GCC_ATOMIC_TEST_AND_SET_TRUEVAL=1 -D__VERSION=\"5.0-13-g4f145b3-master\" -DOMPI_SKIP_MPICXX -std=c++11 -Wall -Wextra -I/apps/all/HDF5/1.14.0-gompi-2022b/include -Isrc -Isrc/Profiles -Isrc/Params -Isrc/Projector -Isrc/Checkpoint -Isrc/picsar_interface -Isrc/ElectroMagnBC -Isrc/ElectroMagn -Isrc/Tools -Isrc/Patch -Isrc/Diagnostic -Isrc/PartCompTime -Isrc/ParticleBC -Isrc/Radiation -Isrc/Merging -Isrc/Interpolator -Isrc/DomainDecomposition -Isrc/Collisions -Isrc/MultiphotonBreitWheeler -Isrc/Pusher -Isrc/MovWindow -Isrc/Field -Isrc/Particles -Isrc/SmileiMPI -Isrc/ElectroMagnSolver -Isrc/Python -Isrc/Ionization -Isrc/Species -Isrc/ParticleInjector -Ibuild/src/Python -I/apps/all/Anaconda3/2023.09-0/include/python3.11 -I/apps/all/Anaconda3/2023.09-0/include/python3.11 -I/apps/all/Anaconda3/2023.09-0/lib/python3.11/site-packages/numpy/core/include -DSMILEI_USE_NUMPY -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -O3 -g -DSMILEI_OPENACC_MODE -DSMILEI_ACCELERATOR_MODE -c src/Checkpoint/Checkpoint.cpp -o build/src/Checkpoint/Checkpoint.o

Edit: only the correct part of the terminal message was kept, so the post is not too long.

[charlesprouveur]

For future reference, this is what i meant:

mpicxx -Wno-reorder -D__GCC_ATOMIC_TEST_AND_SET_TRUEVAL=1 -D__VERSION=\"5.0-13-g4f145b3-master\" -DOMPI_SKIP_MPICXX -std=c++11 -Wall -Wextra -I/apps/all/HDF5/1.14.0-gompi-2022b/include -Isrc -Isrc/Profiles -Isrc/Params -Isrc/Projector -Isrc/Checkpoint -Isrc/picsar_interface -Isrc/ElectroMagnBC -Isrc/ElectroMagn -Isrc/Tools -Isrc/Patch -Isrc/Diagnostic -Isrc/PartCompTime -Isrc/ParticleBC -Isrc/Radiation -Isrc/Merging -Isrc/Interpolator -Isrc/DomainDecomposition -Isrc/Collisions -Isrc/MultiphotonBreitWheeler -Isrc/Pusher -Isrc/MovWindow -Isrc/Field -Isrc/Particles -Isrc/SmileiMPI -Isrc/ElectroMagnSolver -Isrc/Python -Isrc/Ionization -Isrc/Species -Isrc/ParticleInjector -Ibuild/src/Python -I/apps/all/Anaconda3/2023.09-0/include/python3.11 -I/apps/all/Anaconda3/2023.09-0/include/python3.11 -I/apps/all/Anaconda3/2023.09-0/lib/python3.11/site-packages/numpy/core/include -DSMILEI_USE_NUMPY -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -O3 -g -DSMILEI_OPENACC_MODE -DSMILEI_ACCELERATOR_MODE -c src/Checkpoint/Checkpoint.cpp -o build/src/Checkpoint/Checkpoint.o

Because you were using recent module (nvhpc 23.7) you were missing flags such as -gpu=cc70,cc80 -acc etc.

To simplifiy your first compilation, please use nvhpc23.1 and cuda 11.3 as you have these, and have support compile hdf5 with the compiler that comes with it.

Finally your machine file should look like this (i saw that the Karolina cluster is using AMD CPU + A100) :

SMILEICXX.DEPS = nvcc
THRUSTCXX = nvcc

ACCELERATOR_GPU_FLAGS += -w
ACCELERATOR_GPU_FLAGS += -tp=zen3 -ta=tesla:cc80 -std=c++14  -lcurand

ACCELERATOR_GPU_KERNEL_FLAGS += -O3 --std c++14 $(DIRS:%=-I%)
ACCELERATOR_GPU_KERNEL_FLAGS += --expt-relaxed-constexpr
ACCELERATOR_GPU_KERNEL_FLAGS += $(shell $(PYTHONCONFIG) --includes)
ACCELERATOR_GPU_KERNEL_FLAGS += -arch=sm_80
ACCELERATOR_GPU_FLAGS        += -Minfo=accel # what is offloaded/copied 
ACCELERATOR_GPU_FLAGS += -DSMILEI_OPENACC_MODE
ACCELERATOR_GPU_KERNEL_FLAGS += -DSMILEI_OPENACC_MODE

LDFLAGS += -ta=tesla:cc80 -std=c++14  -lcudart -lcurand -lacccuda -L${EBROOTCUDA}lib64/
CXXFLAGS +=  -D__GCC_ATOMIC_TEST_AND_SET_TRUEVAL=1
LDFLAGS = $LDFLAGS:$LD_LIBRARY_PATH
HDF5_ROOT_DIR = ${EBROOTHDF5}

If you want to try nvhpc 23.7, it should look like this:

SMILEICXX.DEPS = nvcc
THRUSTCXX = nvcc

ACCELERATOR_GPU_FLAGS += -w
ACCELERATOR_GPU_FLAGS += -tp=zen3 -gpu=cc80 -acc  -std=c++14  -lcurand

ACCELERATOR_GPU_KERNEL_FLAGS += -O3 --std c++14 $(DIRS:%=-I%)
ACCELERATOR_GPU_KERNEL_FLAGS += --expt-relaxed-constexpr
ACCELERATOR_GPU_KERNEL_FLAGS += $(shell $(PYTHONCONFIG) --includes)
ACCELERATOR_GPU_KERNEL_FLAGS += -arch=sm_80
ACCELERATOR_GPU_FLAGS        += -Minfo=accel # what is offloaded/copied 
ACCELERATOR_GPU_FLAGS += -DSMILEI_OPENACC_MODE
ACCELERATOR_GPU_KERNEL_FLAGS += -DSMILEI_OPENACC_MODE

LDFLAGS += -gpu=cc80 -std=c++14 -acc -cuda  -lcudart -lcurand -lacccuda -L${EBROOTCUDA}lib64/
CXXFLAGS +=  -D__GCC_ATOMIC_TEST_AND_SET_TRUEVAL=1
LDFLAGS = $LDFLAGS:$LD_LIBRARY_PATH
HDF5_ROOT_DIR = ${EBROOTHDF5}

[spadova-a]

Ok, thank you. I will let you know, once the proper HDF5 module is ready and I try the compilation again.

[spadova-a]

Hi,
so I finally got the right HDF5 module available. Nonetheless, I still wasn't successful with the compilation. The latest error is this:
Linking smilei . . . -L/apps/all/HDF5/1.14.0-nvompi-2022.07/lib DFLAGS:D_LIBRARY_PATH -lhdf5 -L/apps/all/Python/3.10.4-GCCcore-11.3.0/lib -lpython3.10 -lcrypt -ldl -lm -lpthread -lutil -lm -lm -Xlinker -export-dynamic /apps/all/binutils/2.38-GCCcore-11.3.0/bin/ld: cannot find DFLAGS:D_LIBRARY_PATH: No such file or directory make: *** [smilei] Error 2
I guess the problem is that it is looking for DFLAGS:D_LIBRARY_PATH instead of LDFLAGS:LD_LIBRARY_PATH. However, I don't know neither why of how to fix it. Any ideas?

[mccoys]

Something is very wrong in your setup. Can you show the result of make env

[iltommi]

also a make config=verbose can help

[spadova-a]

make env:
VERSION : 5.0-57-gc23dd35-master SMILEICXX : mpicxx OPENMP_FLAG : -fopenmp -D_OMP HDF5_ROOT_DIR : FFTW3_LIB_DIR : SITEDIR : /home/spadoalz/.local/lib/python3.10/site-packages PYTHONEXE : python PY_CXXFLAGS : -I/apps/all/Python/3.10.4-GCCcore-11.3.0/include/python3.10 -I/apps/all/Python/3.10.4-GCCcore-11.3.0/include/python3.10 -I/apps/all/Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/numpy/core/include -DSMILEI_USE_NUMPY -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION PY_LDFLAGS : -L/apps/all/Python/3.10.4-GCCcore-11.3.0/lib -lpython3.10 -lcrypt -ldl -lm -lpthread -lutil -lm -lm -Xlinker -export-dynamic CXXFLAGS : -D__VERSION=\"5.0-57-gc23dd35-master\" -DOMPI_SKIP_MPICXX -std=c++14 -Isrc -Isrc/Profiles -Isrc/Params -Isrc/Projector -Isrc/Checkpoint -Isrc/picsar_interface -Isrc/ElectroMagnBC -Isrc/ElectroMagn -Isrc/Tools -Isrc/Patch -Isrc/Diagnostic -Isrc/PartCompTime -Isrc/ParticleBC -Isrc/Radiation -Isrc/Merging -Isrc/Interpolator -Isrc/DomainDecomposition -Isrc/Collisions -Isrc/MultiphotonBreitWheeler -Isrc/Pusher -Isrc/MovWindow -Isrc/Field -Isrc/Particles -Isrc/SmileiMPI -Isrc/ElectroMagnSolver -Isrc/Python -Isrc/Ionization -Isrc/Species -Isrc/ParticleInjector -Ibuild/src/Python -I/apps/all/Python/3.10.4-GCCcore-11.3.0/include/python3.10 -I/apps/all/Python/3.10.4-GCCcore-11.3.0/include/python3.10 -I/apps/all/Python/3.10.4-GCCcore-11.3.0/lib/python3.10/site-packages/numpy/core/include -DSMILEI_USE_NUMPY -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -O3 -g -fopenmp -D_OMP LDFLAGS : -lhdf5 -L/apps/all/Python/3.10.4-GCCcore-11.3.0/lib -lpython3.10 -lcrypt -ldl -lm -lpthread -lutil -lm -lm -Xlinker -export-dynamic -lm -fopenmp -D_OMP COMPILER_INFO : pgc++

and I used the machine file, that was recommended a few comments above

[charlesprouveur]

remove the line:

LDFLAGS = $LDFLAGS:$LD_LIBRARY_PATH

make clean, and try again.
To add some details, LDFLAGS is supposed to be only flags that are added at the linking step, which was redefined in your script with an attempt appending it with the LD_LIBRARY_PATH environment variable. the compiler was looking for -L(OPTION_NAME) and therefore took out the L from LDFLAGS and from LD_LIBRARY_PATH, giving you your error message with "DFLAGS:D_LIBRARY_PATH".

I should have removed this line from your script when i adapted it.