issues with parallel running

[ozgurculfa]

Dear Warpx users,
I am trying to run Warpx parallel but there are some issues appearing and code crushes.
I tried running it with different examples of input. For instance, when I run with plasma_acceleration, it runs but doesn't produce any plot folder(I am not sure if it is supposed to since I am pretty new to the code). But when I run the laser_ion example it crashes giving tons of errors. I have compiled it into two different clusters and it does the same thing in both clusters.
I compiled it from the source file so I am not quite sure what can be the issues. If anyone had that issue before or any idea, I would appreciate of any suggestions.

[ax3l]

Hi @ozgurculfa,

Thanks for reaching out!

Can you share a few details with us on how and where you run?

• system
• module/dependencies
• exact installation you performed (commands and output)
• run scripts with all details

What is the exact runtime behavior?

• output details
• hang / crash details (e.g., backtrace files produced?)

[ozgurculfa]

Hi,
Now, I was trying to recompile with some options and started having issues. I am running them on SDSC Expanse cluster.
https://www.sdsc.edu/support/user_guides/expanse.html#storage
These are the modules I loaded to the system :

1. shared 8) openmpi/4.1.3/oq3qvsv 15) fftw/3.3.10/bmvqsbr
2. cpu/0.17.3b (c) 9) python/3.8.12/7zdjza7 16) hdf5/1.10.7/5o4oibc
3. slurm/expanse/23.02.6 10) py-setuptools/58.2.0/eefp7mw 17) py-mpi4py/3.1.2/kas2whp
4. sdsc/1.0 11) py-numpy/1.20.3/4o6jrav 18) adios2/2.7.1/m6x3v7o
5. DefaultModules 12) perl/5.32.0/aqkfsad 19) openblas/0.3.18/ombsn4y-omp
6. gcc/10.2.0/npcyll4 13) boost/1.77.0/lvxlknh
7. ucx/1.10.1/dnpjjuc 14) cmake/3.21.4/teqow32

I am using
git clone https://github.com/ECP-WarpX/WarpX.git $HOME/src/warpx
cd $HOME/src/warpx
cmake -S . -B build -DWarpX_DIMS="1;2;RZ;3" -DWarpX_OPENPMD=ON -DWarpX_PSATD=ON

And in this part, configuration cant be completed. I am getting this error.

-- Checking for module 'fftw3'
-- Found fftw3, version 3.3.10
-- Found FFTW: /cm/shared/apps/spack/0.17.3/cpu/b/opt/spack/linux-rocky8-zen2/gcc-10.2.0/fftw-3.3.10-bmvqsbr37i4m3556taxaz2bhbvas4uqc
-- FFTW: Could NOT find OpenMP support
CMake Error at CMakeLists.txt:191 (find_package):
Could not find a package configuration file provided by "blaspp" with any
of the following names:

blasppConfig.cmake
blaspp-config.cmake

Add the installation prefix of "blaspp" to CMAKE_PREFIX_PATH or set
"blaspp_DIR" to a directory containing one of the above files. If "blaspp"
provides a separate development package or SDK, be sure it has been
installed.

It looks like it is looking for blasspp but I assumed openblas/0.3.18/ombsn4y-omp module should satisfy it.

Thank you for your time.

[ax3l]

Thanks for the details, @ozgurculfa!

On the SDSC Expanse Cluster, do you aim to compute on the

• Standard Compute Nodes (728 total - CPU only) or
• GPU Nodes (52 total - NVIDIA V100s SMX2 with 4 GPUs per node)
  ?

These are the modules I loaded to the system.

This looks like the CPU partition is targeted for now (let's definitely explore the GPU/CUDA partition as well later, V100s are nice and fast).

The following comments:

• optional: ADIOS2 seems a bit old, you could ask for 2.8.3 or the latest 2.9.1 as well if you like

cmake -S . -B build -DWarpX_DIMS="1;2;RZ;3" -DWarpX_OPENPMD=ON -DWarpX_PSATD=ON

Looks good. The extra dependencies you need if you want to use RZ geometry and the PSATD solver are BLAS++ and LAPACK++.
https://warpx.readthedocs.io/en/latest/install/dependencies.html#install-dependencies

Since you are on a maintained HPC system, you could open a ticket with your system admins and ask for a module for each of them, compatible with your compiler module.

Alternatively, you can install those two modules yourselves, let me know if you like to do that and I provide guidance if needed.

Or, alternatively, you could compile without RZ geometry support or without PSATD solver support, if you do not need one or the other (or neither).

It looks like it is looking for blasspp but I assumed openblas/0.3.18/ombsn4y-omp module should satisfy it.

OpenBLAS is a good dependency to build BLAS++ and LAPACK++ from. Essentially, BLAS++ and LAPACK++ provide us with C++ bindings to BLAS/LAPACK (+ a bit more :) ).

[ozgurculfa]

Thank you so much. I just created a ticket to update adios2 module and lets see how it goes. Once I start running the code with CPU, I might ask for help with GPU :). I am not sure if I will need RZ geometry or PSATD so I will compile and run the code without them first and lets see how it goes :).