More flexible MPI_Reduce_locals needed

[softwaretraff]

Problem

The current MPI_Reduce_local operation (Section 6.9.7 of MPI-4.0) has severely restricted functionality: it "adds" an in-argument to an inout-argument in that order. It is thus not possible to directly "add" two different in-arguments with the result stored in an out-argument, neither is it possible to add two arguments in the order of inout-argument and then in-argument. This limits the effectiveness of MPI_Reduce_local for implementing own, collective reduction operations since it often makes it necessary to copy arguments around.

Proposal

It is proposed to add a 3-argument MPI_Reduce_locals to the standard which by permitting the use of MPI_IN_PLACE provides the full flexibility desirable for implementing own collective reduction operations.

Changes to the Text

MPI_REDUCE_LOCALS( inbuf, argbuf, inoutbuf, count, datatype, op)
IN inbuf input buffer (choice)
IN argbuf input buffer (choice)
INOUT inoutbuf combined input and output buffer (choice)
IN count number of elements in inbuf, argbuf and inoutbuf buf
fers (nonnegative integer)
IN datatype data type of elements of inbuf, argbuf and inoutb
uf buffers
(handle)
IN op operation (handle)

int MPI_Reduce_locals(const void* inbuf, const void argbuf,
void inoutbuf, int count,
MPI_Datatype datatype, MPI_Op op)

MPI_Reduce_locals(inbuf, argbuf, inoutbuf, count, datatype, op, ierror)
TYPE(), DIMENSION(..), INTENT(IN) :: inbuf
TYPE(), DIMENSION(..), INTENT(IN) :: argbuf
TYPE(), DIMENSION(..) :: inoutbuf
INTEGER, INTENT(IN) :: count
TYPE(MPI_Datatype), INTENT(IN) :: datatype
TYPE(MPI_Op), INTENT(IN) :: op
INTEGER, OPTIONAL, INTENT(OUT) :: ierror
MPI_REDUCE_LOCALS(INBUF, ARGBUF, INOUTBUF, COUNT, DATATYPE, OP, IERROR)
INBUF(), INOUTBUF(*)
INTEGER COUNT, DATATYPE, OP, IERROR

The function applies the operation given by op element-wise to the
elements of inbuf and argbuf in that order with the result stored
element-wise in inoutbuf, as explained for user-defined operations in
Section 5.9.5. The inbuf, argbuf and inoutbuf (the two inputs as well
as the result) have the same number of elements given by count and the
same datatype given by datatype. If the MPI_IN_PLACE option is given
for either inbuf or argbuf (or both), the corresponding input is taken
from inoutbuf. The MPI_IN_PLACE is not allowed for the inoutbuf
argument. The inbuf and argbuf buffers are not required to be
distinct, but must be distinct from the inoutbuf argument.

Rationale: In applications (libraries, typically) applying local
reductions with MPI predefined operators, a specific order of the
arguments (for non-commutative, user-defined operators) may be
required, and some argument may not be placed in the required input or
output buffer, which with a restricted, two-argument local reduction
function entails extra, local copying of either or both arguments. The
three-argument function alleviates such extra copying. A call to
MPI_Reduce_local can always be replaced by a call to
MPI_Reduce_locals with MPI_IN_PLACE as the second input
argument.

Examples: Let A, X, Y be the buffers provided for inoutbuf, inbuf, and argbuf,
respectively. The following reduction-assignments can readily be implemented
with MPI_Reduce_locals:

• A = X op Y (with MPI_Reduce_local it would be required to first copy Y into A)
• A = A op Y (if X is MPI_IN_PLACE. With MPI_Reduce_local this would require
  first reducing into Y destructively, and then copying the result Y into A)
• A = X op A (if Y is MPI_IN_PLACE)

and even

• A = X op X (if Y is the same as X)
• A = A op A (if both X and Y are MPI_IN_PLACE)

Impact on Implementations

Implementation should be straightforward in MPI libraries.

Impact on Users

None for existing users. New users will be happy.

References and Pull Requests

[bosilca]

Actually, OMPI does provide a similar capability at a lower level than the MPI API. I mention this in our discussion there, you can find the link above.

[softwaretraff]

Hi George,

that's what I assumed. However, this doesn't help the application programmer. Supporting the 3-argument MPI_Reduce_locals operations should come at very little implementation effort.

Jesper

[softwaretraff]

Hi again,

actually, the discussion in your link is great, and seems to support having a 3-argument MPI_Reduce_locals?

Jesper

[jeffhammond]

Do we need a new MPI_Op_create for this to comply with the C function type?

[softwaretraff]

Good point. Would be possible to do without, but would then cost for user Op's, so perhaps yes

[RolfRabenseifner]

Current MPI_OP_CREATE defines callbacks only for (invec,inoutvec) with elementwise inoutvec[i] = invec[i] o inoutvec[i].
The proposed MPI_REDUCE_LOCALS without MPI_INPLACE therefore always requires
inoutbuf = argbuf ; user_defined_operation(inbuf, inoutbuf )
which is in contradiction to the performance goals of MPI.
May be resolved by providing this new API only for predefinded operations.
But then, a new inquiry function would be needed: MPI_OP_USERDEFINED(IN op, OUT userdefined).

This limits the effectiveness of MPI_Reduce_local for implementing own, collective reduction operations since it often makes it necessary to copy arguments around.

Then the user can implement two different algorithms for predefined and userdefined operations, the first one using the new MPI_REDUCE_LOCALS and the second one usingthe old MPI_REDUCE_LOCAL.

[softwaretraff]

Dear Forum,

I still think a three argument MPI_Reduce_locals as outlined would be tremendously useful for those writing their own library reduction-like functions/collectives - the 2-argument MPI_Reduce_local in many cases forces unnecessary copying, especially if commutativity is not given/to be exploited. The proposal above should be extended by an MPI_Op_create for 3-argument user functions as well. I can provide a proposal/text, if there is interest in taking this to MPI 4.1

Jesper