fix(mpi): ensure wait_any completes at least one request

[dssgabriel]

This PR closes #173 by changing the implementation of Req<Mpi>::wait_any to actually complete at least one request (and calls any registered callbacks on that request) before returning.

Also includes:

• Changed wait to take KokkosComm::Req<Mpi> by reference to avoid a potentially costly deep copy of the postWaits_ callback vector
• Added an overload on wait taking an rvalue-reference so that writing the following remains valid:

  KokkosComm::wait(KokkosComm::send(handle, view, peer));

• Modernized wait_all and wait_any interfaces to use std::span instead of std::vector& references

[cedricchevalier19]

It is semantically correct now but I am not really fond of the active wait. Can't we build a MPI_REQUEST array and use MPI_Waitany?

[dssgabriel]

Sure! I think we should do the same for wait_all in that case 👍

[dssgabriel]

How do we go about calling the registered callbacks on the requests that have completed, though?

[cwpearson]

MPI_Waitany provides the index of the thing that completed, so I think you'd just have to go back and do the callbacks of our associated request.

[dssgabriel]

I changed the implementation to match the MPI_Waitany semantics, i.e., complete one request and exit, as opposed to what I had before, which was more similar to MPI_Waitsome (which may complete multiple requests before returning).

I think we can merge this as it is, and I'll do a follow-up PR that updates our wait_all and wait_any implementation to extract arrays of MPI_Requests and directly call the MPI functions.

[cedricchevalier19]

Does this work for MPI_REQUEST_NULL?
If not, we can end-up with an infinite loop.

[dssgabriel]

I will take a look at what the standard says and fix this accordingly.

[cedricchevalier19]

A call to MPI_TEST returns flag = true if the operation identified by request is complete. In such a case, the status object is set to contain information on the completed operation; if the communication object was created by a nonblocking send or receive, then it is deallocated and the request handle is set to MPI_REQUEST_NULL. The call returns flag = false, otherwise. In this case, the value of the status object is undefined. MPI_TEST is a local operation.

I am not an MPI expert but my understanding is that flag = false if the input is MPI_REQUEST_NULL as it cannot be completed.