Debugging and building different MPI communication & decomposition libraries

[aronroland]

This Idea thread is about providing a unified developing enviroment in terms of usage, compilation and debugging MPI code.

[aronroland]

The 1st part is linked to the intel mpi library, which ist the most dominant in most of the HPCF environments and comes with the compiler itself.

Since impi (intel mpi) is not compiled by the user/admin there are no build options.

Intel controls the debug output by applying environmental variables that are defined here:

"https://www.intel.com/content/www/us/en/docs/mpi-library/developer-guide-linux/2021-6/displaying-mpi-debug-information.html"

Various debug levels are available, which helps the ww3 development team debug mpi errors.

[aronroland]

The 2nd part is about MPICH (... to be continued)

[aronroland]

3rd part is about OpenMPI (... to be continued)

[aronroland]

Parallel decomposition libraries.

When running unstructured grid models using the domain decomposition (DD) concept for parallelization a graph partitioning software is needed. Actually, there are two options that are available in WW3.

1. SCOTCH (This is the new default DD library)
2. ParMetis (This was the former default DD library)

The compilation of those libraries should be done on each HPCF system, where WW3 is implemented, in two settings. The 1st setting should be for debugging an analysis production and the 2nd setting should be for debugging. As WW3 is a community effort the latter can significantly speed up development and error handling from the side of the developers and helps identify and solve potential and future issues.

1. Building the SCOTCH library

1.1 Debug mode

At this level, even if cmake is now in active development for the SCOTCH library we remain with the GNU makefile system, which is located in scotch/src/Make.inc directory.

In this directory u can find the following makefile that need to be copied from this directory in the src directory and renamed to Makefile.inc.

For debugging of SCOTCH we pick at this level the following makefile "Makefile.inc.x86-64_pc_linux2.icc.impi.debug". The compilation is just done by typing "make".

1.2. Performance model

For performance builds we have chosen that "Makefile.inc.x86-64_pc_linux2.icc.impi" has to be used.

1.3. Instrumenting SCOTCH for more debug info for the SCOTCH team

In order to add cpp Pragma for the sake of instrumenting further debug info and in-depth debugging information for the SCOTH team, the various debug flags can be added to l.13 of "Makefile.inc.x86-64_pc_linux2.icc.impi.debug". There are already several debug options added.

3. Building the ParMetis library (... to be continued)

[MatthewMasarik-NOAA]

1.3. Instrumenting SCOTCH for more debug info for the SCOTCH team

In order to add cpp Pragma for the sake of instrumenting further debug info and in-depth debugging information for the SCOTH team, the various debug flags can be added to l.13 of "Makefile.inc.x86-64_pc_linux2.icc.impi.debug". There are already several debug options added.

Hi @aronroland, thank you this is indeed very helpful. I was able to compile on our hera machine using the Make profile you listed. For this machine I did need one fix to complete the build. In the file, src/libscotch/common_integer.c, I added the following directives at the top of the file to overcome a PRIu64 related build error:

#define __STDC_FORMAT_MACROS
#include <inttypes.h>

With that the SCOTCH build succeeds. I get an error when I then try to compile WW3 with that SCOTCH build, which I'm working to sort out now.

[aronroland]

@MatthewMasarik-NOAA , "slowly the squirrel feeds" is a saying in Germany but finally it survives the winter. We need now just to make sure that we are carefully defining standards and following the quest. Thanks @MatthewMasarik-NOAA checking on that one ...

[MatthewMasarik-NOAA]

Hi @aronroland, what an interesting nugget! I sadly have not been to Germany, so I'm not certain if I understand what is meant.. Can you please help me understand what you intended to share in this case?

[aronroland]

... it means something like if u work eagerly and put all pieces together slowly and carefully u will reach your goals ...

[MatthewMasarik-NOAA]

Hi @aronroland , thank you for clarifying the message. I do like the imagery, and it is nice general sentiment. I wonder if it's perhaps a poor analogy for the circumstances though? When I think of the amount of work done on this investigation so far, and the intensity, I see this as a very active, deliberate attack on a problem. I unfortunately could not include all the details in the notes I gave, so this is probably why you are unaware. I'm happy to discuss details anytime to help get you up to speed, so let me know if interested.