Reductions on REAL*16's are not working

[ompiteam]

Per a thread on the user's list:

http://www.open-mpi.org/community/lists/users/2008/10/7081.php

Something odd is going on with REAL*16 reductions. George and I have discovered:

• The MPI_REAL16 datatype is mapping itself onto the MPI_LONG_DOUBLE datatype (which is fine)
• So its actually invoking ompi_mpi_op_sum_long_double() to do the reduction (instead of ompi_mpi_op_sum_fortran_real16()), but this should be ok because the ompi_fortran_real16_t is just a typedef for long double. So these functions are identical (but it does mean that we have a few useless op routines).
• What it ''looks'' like is that the DDT engine is copying over the last element as the first step of the reduction, but then the "+=" operator is not actually adding in the other values together.
• Ditto for other operators (e.g., MAX). It's like the operators are not having any effect.

Very puzzling.

[ompiteam]

Imported from trac issue 1603. Created by jsquyres on 2008-10-28T16:16:56, last modified: 2011-06-18T21:12:17

[ompiteam]

Trac comment by jsquyres on 2008-10-28 16:58:20:

Wow, this is a fun one. George and I discovered that icc is representing "long double" in 81 bits, but ifort represents "REAL_16" in all 128 bits. Currently, OMPI's op routines are all implemented in C, and therefore it's just doing the Wrong Thing for Fortran REAL_16.

George and I are discussing a few options for fixing this -- it's debatable as to whether this will get into v1.3.0. It's very unlikely that we'll fix v1.2.

[ompiteam]

Trac comment by bosilca on 2008-10-28 17:37:39:

I went a little bit further. Apparently the REAL16 type supported by the Intel Fortran compiler use a different data representation than long double (both use the standard ''IEEE754'', but the size of the significant is larger and the exponent is not on the same bits). Based on some web resources (such as [http://www.cactuscode.org/pipermail/developers/2006-April/004658.html CactusCode]) it appears that the 128 bits REAL16 is software emulated on most platforms as there is no hardware support for this type. This [of course] incur a significant performance penalty.

I hardly imagine adding such a feature in Open MPI. Yes it give a better precision but it greatly affect performance. Moreover, based on discussions with the math guys from here:

If you go over 64 bits for floating point computations is that your problem is ill conditioned, and you better fix the conditioning and go with a standard and faster type such as long double (i.e. REAL8).

What I propose is to fix this at the '''configure''' level. In Fortran we can use the function precision(TYPE) to figure out the precision of the Fortran data representation. The standard precision for long double is 10. If we get anything bigger than this (as an example ifort report 16 for REAL16), we completely disable the support for REAL16. Easy, fast and can even be back-ported to the 1.2

[ompiteam]

Trac comment by jsquyres on 2008-10-28 18:02:05:

Wouldn't it be better to find out if there's a mismatch between long double and REAL_16, rather than checking against an arbitrary value (10)? If there's a mismatch, disable support for REAL_16.

E.g., we could do some simple MPI_SUM-like operation in a configure test and see if we come up with the right answer. If we don't, disable REAL*16.

[ompiteam]

Trac comment by jsquyres on 2008-10-29 10:22:14:

George and I talked more off-ticket. Some notes:

• Intel does have an option for making C and Fortran types the same. Use the "_Quad" type in C and use the following compiler flags: -Qoption,cpp,--extended_float_types.
• Pathscale doesn't support REAL*16
• GNU doesn't support REAL*16
• PGI supports REAL_16, but their C long doube bit representation is also different than their Fortran REAL_16 representation. I didn't look further to see if there's a C compiler type/option to make them the same (like Intel). Assumedly, there is.
• There are three options:
  1. Disable all support for MPI_REAL16
  2. Leave message passing support for MPI_REAL16 but disable reduction support
  3. Fix reduction support for MPI_REAL16
• It seems like the "fix" option is going to take a while, so we're going to defer it for now. This ticket will stay open to until the issue is resolved one way or another.
• The workaround for now seems to be to disable MPI_REAL16 support for reductions, and advise application authors to either use more limited precision (because the performance of REAL16 is going to be bad anyway) or use user-defined Fortran MPI operators to do their own reductions on MPI_REAL16 types.

[ompiteam]

Trac comment by tdd on 2008-10-29 14:46:48:

I've sent Sun's compiler people the question about the Real*16 vs long double support and I got the following reply:

On SPARC they are the same. On x86, we are adding support for REAL_16. Once that is done, the layout will be different. C's long double is actually an 80 bit number, whereas Fortran's REAL_16 is a 128 bit number.

So, With the above it might seem nice to have MPI_REAL16 reductions be configurable.

[ompiteam]

Trac comment by jsquyres on 2008-10-29 14:52:41:

I've made an hg tree to do this work:

http://www.open-mpi.org/hg/hgwebdir.cgi/jsquyres/real16/

Three things need to be done:

1. Make MPI_REAL16 and MPI_COMPLEX32 no longer be aliases; they need to be their own types (so that MPI_Op can key off them as a different case than MPI_LONG_DOUBLE)
2. Add a configure test/option to enable/disable reductions for REAL16 and COMPLEX32
3. Adjust the MPI_Op predefined operations for REAL16 and COMPLEX32 to be enabled or disabled

George: can you do item 1?

I can do item 2 and item 3.

[ompiteam]

Trac comment by tdd on 2008-10-30 08:38:25:

Upon George's request I've ask our compiler group how 128 bit mathematics are supported and what C-Type corresponds to Real*16.

The answer I got was both SPARC and x86 128 bit mathematics are supported via software. As to the C-Type question for x86 there are none that corresponds to Real_16 but as one can expect (and mentioned in my previous message) on the SPARC platforms long double does correspond to Real_16.

[ompiteam]

Trac comment by jsquyres on 2008-10-30 16:27:34:

HG tree now has configure support and op.c is fixed. George -- the macro you want is OMPI_REAL16_MATCHES_C. It'll always be 0 or 1.

Throwing this ticket to George for completion...

[ompiteam]

Trac comment by bosilca on 2008-11-06 17:19:47:

This is my worst nightmare. Allowing the '''REAL16''' and '''COMPLEX32''' to be used for pt2pt messaging, is still kind of tricky. Having such a datatype, will allow the user to use it for creating others datatype, and use them in file operations or such.

Therefore, adding support for such types everywhere except the op is a difficult topic. There are two other functions that need to be modified: mpi/c/type_create_f90_complex.c and mpi/c/type_create_f90_real�.c. And there I don't have any clues what the ''number of digits'' and the ''max_10_exp'' is for such obscure F90 types.

How do we stop the user from using such a type in creating other types ? How do we convert such a type from one architecture to another on a heterogeneous case ?

To be honest this type is so ugly, that the only thing I wish is to completely drop support for these types. The user will still be able to use it, by creating their own type based on a number of MPI_BYTE. With such an approach, no op will be allowed, except if the user define them, no conversion is possible and so on.

[ompiteam]

Trac comment by jsquyres on 2008-11-06 20:29:45:

I talked with George a bit more extensive about this on the phone and I've become convinced that he's right. What we should do for 1.3.0 is only enable MPI_REAL16 support on platforms where long double === REAL*16 (i.e., both size and bit representation), such as SPARC.

If we enable MPI_REAL16 where the C and Fortran bit representations don't match, problems will occur with the following:

• Reductions (obviously)
• MPI_PACK_EXTERNAL
• Heterogeneity, in terms of point-to-point and MPI I/O operations
• ...and probably some others

George will update the HG tree to do the Right Thing in the DDT engine when sizeof(long double) == sizeof(REAL*16) ''and'' OMPI_REAL16_MATCHES_C is set to 1. Then we'll be good to go for 1.3.0.

In future versions, we can investigate using special C compiler types to match the representation of REAL*16 (like _Quad in the intel compiler).

[ompiteam]

Trac comment by jsquyres on 2008-11-07 15:37:23:

(In [19948]) Refs https://svn.open-mpi.org/trac/ompi/ticket/1603:

• Add OMPI_F77_CHECK_REAL16_C_EQUV test whether REAL*16 is bit
  equivalent to long double. AC_DEFINE OMPI_REAL16_MATCHES_C with
  result (0 or 1).
• Update ompi_info to only show real16 support if
  OMPI_REAL16_MATCHES_C is 1.
• Update DDT to only support REAL16 and COMPLEX32 if
  1==OMPI_REAL16_MATCHES_C.
• MPI Op function pointer tabls will have NULL for the REAL16 and
  COMPLEX32 entries if 0==OMPI_REAL16_MATCHES_C.
• Slightly cleaned up OMPI_F77_GET_ALIGNMENT and OMPI_F77_CHECK m4
  tests (use OMPI_VAR_SCOPE_PUSH/POP).