MPI_Comm_idup*: allow copying of new comm handle before completion

[jprotze]

Problem

The standard states:

It is erroneous to use the communicator newcomm as an input argument to other MPI functions before the MPI_COMM_IDUP operation completes.

But there is no text that clarifies, whether the MPI_Comm variable referenced by newcomm can be copied (e.g., passed by value to a function call) before completion. As request is mentioned in the reference to nonblocking collective semantics, but newcomm is not, the delayed modification of the handle value seems not be covered.

In general, it would be possible that an implementation stores the address and fills the variable during operation completion, making a copy of the MPI_Comm variable invalid.

Proposal

Add text to clarify whether the content of the newcomm variable can be safely copied after return of MPI_COMM_IDUP(_WITH_INFO) or add a restriction.

Changes to the Text

TBD

Impact on Implementations

OpenMPI and MPICH set the handle before return of the function. They also copy attributes of the old communicator to the new communicator before returning, as required by the standard. With the requirement to copy attributes before return, it does not really make sense to delay setting the newcomm variable until the operation completes.

Impact on Users

Be certain to write correct code :)

Impact on Tools

Be able to track communicator creation inside the non-blocking call without the need to keep track of the request and collecting the handle information on operation completion.

References and Pull Requests

[tonyskjellum]

I think that newcomm is simply unusable till completion, and that is that :-) Making implementations promise certain levels of use of the handle before completion is possibly expedient but I think not a good idea. I agree you don't know if you can make copies of the handle before completion. I would say not.

…

On Wed, Aug 24, 2022 at 10:51 AM Joachim ***@***.***> wrote: Problem The standard states: It is erroneous to use the communicator newcomm as an input argument to other MPI functions before the MPI_COMM_IDUP operation completes. But there is no text that clarifies, whether the MPI_Comm variable referenced by newcomm can be copied (e.g., passed by value to a function call) before completion. As request is mentioned in the reference to nonblocking collective semantics, but newcomm is not, the delayed modification of the handle value seems not be covered. In general, it would be possible that an implementation stores the address and fills the variable during operation completion, making a copy of the MPI_Comm variable invalid. Proposal Add text to clarify whether the content of the newcomm variable can be safely copied after return of MPI_COMM_IDUP(_WITH_INFO) or add a restriction. Changes to the Text TBD Impact on Implementations OpenMPI and MPICH set the handle before return of the function. They also copy attributes of the old communicator to the new communicator before returning, as required by the standard. With the requirement to copy attributes before return, it does not really make sense to delay setting the newcomm variable until the operation completes. Impact on Users Be certain to write correct code :) Impact on Tools Be able to track communicator creation inside the non-blocking call without the need to keep track of the request and collecting the handle information on operation completion. References and Pull Requests — Reply to this email directly, view it on GitHub <#577>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAG7ZCI4G5G76BJHS57RBLTV2YZIVANCNFSM57PU44NQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

-- Anthony Skjellum, PhD ***@***.*** Cell: +1-205-807-4968

[wgropp]

In the case of other objects being filled in by a nonblocking operations, those objects cannot be accessed until the operation completes. We had a long discussion about this for Isend and Irecv. I see no reason to make this different. Until the request completes, the newcomm must not be accessed, even to copy it into another variable. We could add a reminder about this - I do not think it would be a change.

[jprotze]

I would argue, that the opaque handle is a reference to the implementation's internal data structure. It is clear, that the data structure is not usable before completion, but some information must be stored anyways (like a copy of the attributes at the time of the non-blocking function call!). Allowing to copy the handle value would require the implementation to setup the mapping of handle value to internal data structure at the time of the non-blocking call.
Similarly, the buffer address used in Isend/Irecv is a reference to the communication buffer. Nothing is wrong with passing around the buffer address and accessing the buffer using a copy of the address after completion.

For a correctness analysis tool like MUST, this change would allow to actually diagnose and report the use of such handle before completion. Without this change, the tool can only setup the mapping for the handle at completion. So, a use of the handle before completion would be reported as "use of unknown handle" rather than "use of incomplete handle created by Comm_idup".

[wgropp]

I think this argument is putting too much emphasis on the standard - just as has often been done for error handling. Just because the MPI standard doesn't define exactly what happens does not prohibit a tool from exploiting knowledge about particular implementations. Yes, having the standard be specific gives the tools an extra level of confidence - but they can check the MPI version string and decide what they can do. And as noted, the implementations are unlikely to change - just as a "bad rank" error return is unlikely to leave any real MPI implementation in a damaged state.

[wesbland]

This wasn't read at the December 2022 meeting. The last opportunity for MPI 4.1 is to have it ready at the March 2022 meeting (and it needs to "pass" the reading).

@tonyskjellum Are you (or is someone else) planning to push this forward or should we move it out of the plan for MPI 4.1?