MPI_Reduce(3) man page: output buffer lengths on non-root ranks

[nathanweeks]

The MPI_Reduce(3) man page in Open MPI 2.1.0 states:

The routine is called by all group members using the same arguments for count, datatype, op, root, and comm. Thus, all processes provide input buffers and output buffers of the same length, with elements of the same type.

This corresponds to the wording in the MPI 2.2 standard. However, the MPI 3.0 wording was changed to:

The routine is called by all group members using the same arguments for count, datatype, op,
root and comm. Thus, all processes provide input buffers of the same length, with elements
of the same type as the output buffer at the root.

Does the MPI 3.0 wording legalize the use of a recvbuf with, e.g., length == 0 for non-root ranks, and if so, does the Open MPI implementation of MPI_Reduce allow this (in which case, the Open MPI MPI_Reduce(3) man page should be updated)?

[bosilca]

Why don't you simply use NULL as the recv buffer on non-root processes ? You can't use 0 for count as it also applies to the send buffer, and they all need to participate with something.

As an example you could do:

#include <mpi.h>
#include <stdio.h>
#include <string.h>
#include <stdlib.h>

#define LENGTH (1024*1024)

int main( int argc, char* argv[] )
{
    int i, rank, size, root = 0, errors = 0;
    int *buf = (int*)malloc(LENGTH * sizeof(int));

    MPI_Init(NULL, NULL);

    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);

    for(i = 0; i < LENGTH; buf[i] = rank, i++);
    if( root == rank )
        MPI_Reduce(MPI_IN_PLACE, buf, LENGTH, MPI_INT, MPI_MAX, root, MPI_COMM_WORLD);
    else
        MPI_Reduce(buf, NULL, LENGTH, MPI_INT, MPI_MAX, root, MPI_COMM_WORLD);

    if( root == rank ) {
        for( i = 0; i < LENGTH; i++ ) {
            if( buf[i] != (size-1) ) {
                printf("error at position %d: %d != %d (found != expected)\n",
                       i, buf[i], size);
                errors++;
            }
        }
        if( 0 == errors ) {
            printf("All good (reduce with NULL recvbuf on non-root processes supported).\n");
        }
    }
    MPI_Finalize();
    return (0 == errors ? 0 : -1);
}

[nathanweeks]

I was careful to refer to the "length" of recvbuf being 0, rather than the count argument of MPI_Recv(); e.g., the following Fortran program is syntactically convenient in that it has only one call to MPI_Reduce(), rather than different calls with different argument lists for for rank == 0 and rank /= 0, reducing the opportunity for me to make a mistake in the argument list (although the dummy argument names and better type checking facilitated by the mpi_f08 module has certainly helped with that).

program mpi_reduce_sum
   use mpi_f08
   implicit none

   integer, parameter :: N = 2**30
   integer :: nproc, rank
   integer, allocatable :: sendbuf(:), recvbuf(:)

   call MPI_Init
   call MPI_Comm_size(comm=MPI_COMM_WORLD, size=nproc)
   call MPI_Comm_Rank(comm=MPI_COMM_WORLD, rank=rank)

   allocate(sendbuf(N), source=1)
   if (rank == 0) then
      allocate(recvbuf(N))
   else
      allocate(recvbuf(0))
   end if

   call MPI_Reduce(sendbuf=sendbuf, recvbuf=recvbuf, count=size(sendbuf),datatype=MPI_REAL,op=MPI_SUM,root=0,comm=MPI_COMM_WORLD)

   if (rank == 0) write(*,*) merge('PASS', 'FAIL', all(recvbuf == nproc))
    
   call MPI_Finalize
end program mpi_reduce_sum

While the use case I had in mind comes involves Fortran allocatable arrays, it should be applicable to C/C++, e.g., using NULL pointers.

[bosilca]

The length of an array is something that only makes sense in Fortran, and Fortran has not been mentionned on the original text so I assumed you were talking about the count (which is the only "length" information that C knows).

Anyway, just to be clear non-root processes completely ignore the recv buffer.

[nathanweeks]

Got it. So it seems like this part of the the Open MPI MPI_Reduce(3) man page would be more accurate if updated to reflect the MPI 3.[01] wording?

[bosilca]

👍