All tests for long integers segfaults

[davidedelvento]

I'm trying to compile ScaLAPACK with long (8-byte) integers as described in the README. A complication is that I am doing so within spack, as described at spack/spack#49950

All the tests crash with segmentation fault. As I wrote in the link above, I am not really sure if the culprit is a missing +ilp64 in a dependency (*), the way I configured spack to build things, my own specific environment or something else.

Another complication is that in that environment I am using gcc/14.2.0 which is not yet supported here (but I believe that spack adds a patch to address than and it also applies a patch for #23 even though that was rejected here).

So I guess my question is if anybody is able to confirm that v2.2.0 and newer of ScaLAPACK with long (8-byte) integers compiles and passes the tests (not all of them, but at least the vast majority as described in the README) at least with gcc v13 (I can't easily do such a thing myself since I would need to recompile OpenMPI and many other dependencies) and does not segfaults anything.

TIA
(*) if I intentionally link with openblas without +ilp64 contrary to the README instructions, then the segmentation faults are reduced to only 74 rather than the several thousands I get when doing it "right"

[zerothi]

I believe this also requires an MPI compiled with long int-support. So that might be the reason?
Default OpenMPI uses int, not long.

[davidedelvento]

Thanks @zerothi that is exactly what I thought myself too!!
But digging deeper I found #83 and my understanding of it seemed to suggest that compiling MPI with long-int was not possible and perhaps not needed? Hence my question if anybody was able to get ilp64 in Scalapack to ever work, and how....

[davidedelvento]

Okay, to answer my own question, jumping through some hoops, I was able to build with ilp64 as follows

FC=mpif90 FFLAGS="-O3 -fdefault-integer-8 -Wno-error=implicit-function-declaration -fallow-argument-mismatch" CFLAGS="-O3 -DInt=long -Wno-error=implicit-function-declaration -std=gnu89" cmake -DCMAKE_LINKER=/home/sw/other/compiler_wrapper/1.3/ld .. && make -j 32

This was with gcc/14.2.0 and against openmpi/5.0.5 and openblas/0.3.28-ilp64.
The FC assignment was needed otherwise it would use plain gfortran to link and fail to find the needed MPI symbols at link stage.

After building that way, surprisingly enough, all tests ran without even a single segfault. Even more surprisingly, xssep, xsgsep and xssyevr passed!

Instead, 83 tests of xcheevr failed and 54 passed (none were skipped).

So the question is if anybody can reproduce these results and confirm that the expected failures for xssep, xsgsep and xssyevr are not expected anymore, and instead there are expected failures for xcheevr. Or disprove this last sentence.

[davidedelvento]

@albandil can you please confirm that for scalapack v2.2.0 or newer, following the instructions of your PR#19 as described in my earlier comment still has the expected failures for xssep, xsgsep and xssyevr and no expected failures for xcheevr? TIA!

[albandil]

@davidedelvento I tried the test suite with the current git master, GCC 14.2, OpenMPI 5.0.7 (default configuration), and with OpenBLAS 0.3.29 (ILP64). The flags that I used to compile ScaLAPACK were

    -D CMAKE_C_FLAGS="-DInt=long -Wno-implicit-function-declaration" \
    -D CMAKE_Fortran_FLAGS="-fdefault-integer-8 -fallow-argument-mismatch" \

The outcome was the following:

The following tests FAILED:
          1 - xCbtest (Failed)
          2 - xFbtest (Failed)
         77 - xssep (Failed)
         81 - xsgsep (Failed)
         91 - xssyevr (Failed)
         95 - xshseqr (Failed)
         96 - xdhseqr (Failed)
Errors while running CTest

The first test was killed by me becuse it hangs somewhere, the rest failed on their own. I suspect that xshseqr and xdhseqr, which are new on this list for me, may be failing due to insufficient workspace. I will check all of the above later today.

EDIT:

Right, so the two tests xshseqr and xdhseqr were failing for me even with the default 4-byte integers, and only when I built ScaLAPACK in the Debug mode. This is unrelated to the ILP64 mode. Avoiding needless divisions by a zero integer in two places, which are not normally performed if reduced condition evaluation takes place, was enough to get around that:

diff --git a/SRC/pdtrord.f b/SRC/pdtrord.f
index 63e9457..99114e9 100644
--- a/SRC/pdtrord.f
+++ b/SRC/pdtrord.f
@@ -1085,7 +1085,7 @@
                END IF
 *
                IF( FLOPS.NE.0 .AND.
-     $              ( FLOPS*100 ) / ( 2*NWIN*NWIN ) .GE. MMULT ) THEN
+     $              ( FLOPS*100 ) .GE. MMULT * ( 2*NWIN*NWIN ) ) THEN
 *
 *                 Update off-diagonal blocks and Q using matrix-matrix
 *                 multiplications; if there are no Householder
diff --git a/SRC/pstrord.f b/SRC/pstrord.f
index 6960914..8c8dcb1 100644
--- a/SRC/pstrord.f
+++ b/SRC/pstrord.f
@@ -1085,7 +1085,7 @@
                END IF
 *
                IF( FLOPS.NE.0 .AND.
-     $              ( FLOPS*100 ) / ( 2*NWIN*NWIN ) .GE. MMULT ) THEN
+     $              ( FLOPS*100 ) .GE. MMULT * ( 2*NWIN*NWIN ) ) THEN
 *
 *                 Update off-diagonal blocks and Q using matrix-matrix
 *                 multiplications; if there are no Householder

The tests xCbtest and xFbtest were failing for two reasons. First, they are written with the assumption that the default Fortran integer and the MPI Fortran API integer are the same: MPI_Type_size calls are used to query sizes of integers using MPI_INTEGER. This breaks down when ScaLAPACK is compiled with -fdefault-integer-8 but Open MPI is not. As far as I know, these two tests are the only places where the mismatch matters. Otherwise MPI is called from C, bypassing the MPI Fortran API altogether. I was able to fix this for a long-integer build by changing MPI_INTEGER to MPI_INTEGER8 in "BLACS/TESTING/btprim.f". I also had to include -finit-local-zero among the Fortran compiler flags, because the default 4-byte MPI Fortran interface obviously sets only the lowest four bytes of any long integer passed in (like RANK or NPROC) and leaves the higher four bytes uninitialized. Then I got again down to the following list mentioned in #19:

The following tests FAILED:
         77 - xssep (Failed)
         81 - xsgsep (Failed)
         91 - xssyevr (Failed)
Errors while running CTest

Alternatively, using Open MPI compiled with 8-byte integer interface also fixes the tests xCbtest and xFbtest, without the need of the above-mentioned tweaking of the code and of the compiler options. It may not be the case for other implementations of MPI. (I believe that the ILP64 version of Intel MPI, for instance, still maintains equivalence between MPI_INTEGER and MPI_INTEGER4.)

I did not try building the software with Spack.

[davidedelvento]

Thank you so much @albandil this is really helpful. I was indeed doing a couple of things slightly differently, but then doing it in the same way as you do I obtained (almost) the same results. I say almost because I get slightly different numbers. Being smaller than yours I can be content and close this ticket.

The following tests FAILED:
         51 - xssep (Failed)
         55 - xsgsep (Failed)
         65 - xssyevr (Failed)
Errors while running CTest

I will work with the spack community to debug what I may have done wrong there and fix it. Thanks again!

[sobukus]

What's the current state now? I also tried defining Int to long (would prefer int64_t …) and setting default Fortran integer to 8 and got segfault all around in MPI (which is using 4-byte integers). Is Scalapack supposed to safely use 32 bit MPI integers now (packing large arrays)? Is it requiring a ilp64 MPI? Would be nice to have the README updated. The known failing tests might also be just fixed?

The demand for 64 bit indices in BLAS stuff from our users is getting more and more real.

[davidedelvento]

@sobukus if you are using spack, make sure to pull in scalapack v2.2.2 not v2.2.0.

If you are not using spack, things should work okay as described in the README and @albandil comment.

In both cases, the only problem is MPI, as you say is MPI. The MPI standard has not allow 64 bit integers, period. If you really and absolutely need long integers in MPI, see #83 for (painful) workaround.

[sobukus]

@davidedelvento Well, yes, MPI is a problem, because of that I observe the segfaults like the initial poster. The README only talks about the two failing tests that probably just should be fixed. @albandil describes a little journey that could benefit from unfolding. I don't get it how only the two tests have places where the mismatch between Int and MPI integer (4 byte) matters — why are all the other tests segfaulting in MPI? Also @albandil built with a 8-byte-integer MPI library in the end, or not?

What about the fork by AMD (https://github.com/amd/aocl-scalapack/)? It claims easy ILP64 support. I don't see ILP64 MPI being mentioned. Maybe there could be work merged back? Obviously they did not only work on AMD Zen optimizations.

[albandil]

Hi @sobukus!

I also tried defining Int to long (would prefer int64_t …) and setting default Fortran integer to 8 and got segfault all around in MPI (which is using 4-byte integers).

If by "segfault all around" you mean segfaults in a large portion of the ScaLAPACK test suite, then this is quite surprising to me. Both the original poster and I see only a few failures. Did you also apply the necessary replacements in the test sources as mentioned in the README file?

I agree with int64_t being preferable. But if I recall correctly, this might require extra #include <stdint.h> here and there with certain compilers.

Also @albandil built with a 8-byte-integer MPI library in the end, or not?

I do not use 8-byte-Fortran-integer MPI library. I may have tried using it when I was contributing the ILP64 functionality to this repository, but I am not sure if it works. In any case, the current README file does not mention the need of 8-byte-Fortran-integer MPI and it should not need it.

I don't get it how only the two tests have places where the mismatch between Int and MPI integer (4 byte) matters

If you refer to xCbtest and xFbtest, then the issue is that they call the Fortran subroutine IBTSIZEOF, which is not used anywhere else in the library. This code uses MPI_INTEGER to communicate Fortran default integer arrays. But MPI_INTEGER should be replaced with MPI_INTEGER8 if -fdefault-integer-8 was used.

If I grep for MPI_INTEGER in the current code of ScaLAPACK, then I see it being used only in the sources for xCbtest and xFbtest (specifically, file btprim.f), and then in the auxiliary program fmpi_sane.f. The latter is a separate case that quite likely "just works" on little-endian systems regardless of the integer size, as it communicates only one such integer containing a small number, so it does not matter if only half of the memory is communicated.

What about the fork ...

I agree that there are multiple opportunities for improvement to make the tests work out of the box. Some of the tests could be fixed simply by using the Fortran intrinsic storage_size, for instance.

[sobukus]

I do mean all the tests (or nearly all, at least), as the initial post here also implied. @davidedelvento then later changed the build approach (from spack?) and got only the few failures. I think I did what he did later, with the Int define and default size 8. I'll have to check my setup again, maybe my MPI crashed for another reason, then.

But, judging from #83 , is the ilp64 scalapack really safe regarding dimensions > 32 bits that could end up as integer counts in MPI calls? The MPI_INTEGER8 thing is only about the data type, not the counts, right? Do we need the wrappers discussed in #83 to have actually 64-bit-safe scalapack? Wouldn't it just crash/miscompute otherwise if we trigger an overflow of an integer counter in MPI_Send()? I wonder if the AOCL fork has that handled.

[albandil]

But, judging from #83 , is the ilp64 scalapack really safe regarding dimensions > 32 bits that could end up as integer counts in MPI calls? The MPI_INTEGER8 thing is only about the data type, not the counts, right? Do we need the wrappers discussed in #83 to have actually 64-bit-safe scalapack? Wouldn't it just crash/miscompute otherwise if we trigger an overflow of an integer counter in MPI_Send()? I wonder if the AOCL fork has that handled.

Yes, absolutely. If a need arises in ScaLAPACK to actually send a number of elements greater than INT_MAX, then this is a problem. The ILP64 extension that is presently implemented here is particularly motivated by the need to be able to allocate large memory blocks for large matrices and to perform indexing within such blocks. Given that ScaLAPACK does not typically send all process's data at once in MPI calls but rather a subset at a time, this is often all that is needed.

Support for large counts in the underlying MPI communication might be useful. I am not sure that AOCL has done any major advancements in that direction, though. On the one hand, they seem to call MPI routines as usual, see for example https://github.com/amd/aocl-scalapack/blob/master/BLACS/SRC/BI_Ssend.c (no parititioning of a potential large message is implemented e.g. when called from https://github.com/amd/aocl-scalapack/blob/master/BLACS/SRC/dgesd2d_.c), on the other hand they seem to explicitly imply 32-bit MPI C integers in their in-house code, see for example amd@d3f06a4#diff-82b47969a7cbd5840cba2953a93f04205719e84cae2f4fe8abc0e0331509662b (an Int array is changed to an int array for use in an MPI call). But I looked at the repository only very superficially.

A systematic approach suggested in #83 for overflow checking looks like a great initiative. Without that, potential integer overflows in huge MPI calls will indeed lead to undefined behaviour like crashes or silent truncations.