Info query

[dholmes-epcc-ed-ac-uk]

Pull request:
https://github.com/mpi-forum/mpi-standard/pull/27

Problem

Context: MPI_Comm/File/Win_get_info

MPI_COMM_GET_INFO returns a new info object containing the hints of the communicator associated with comm. The current setting of all hints actually used by the system related to this communicator is returned in info_used. If no such hints exist, a handle to a newly created info object is returned that contains no key/value pair. The user is responsible for freeing info_used via MPI_INFO_FREE.

What does “actually used” mean?

• Info settings exactly as supplied by the user (e.g. during creation or in set_info)
• Current effective values of info keys (even if different/not supplied by user)
• Implementation defined (this is the most common current interpretation)

Also, when can the value for an info key change?

• Whenever the MPI library feels like it?
• Only in user calls that access/set a particular info key?

Proposal

• MPI maintains an internal INFO state for each Comm/File/Win object.
  • Different INFO state for each object.
  • The INFO state is created/set/referenced during creation of the Comm/File/Win object.
  • The INFO state may be INFO_NULL, if no keys support by MPI.
  • The INFO state may be some common default Info object, if no user-supplied key/values.
• MPI_[Comm|File|Win]_get_info returns MPI’s current internal INFO state.
• MPI_[Comm|File|Win]_set_info attempts to modify MPI’s internal INFO state.
  • MPI may ignore supplied keys, whether or not that key is current set in the internal state.
  • MPI may choose different values for supplied keys, (not user value and not internal value).
  • MPI must not change values for keys not supplied by the user in the set function call.
  • MPI must not choose more restrictive values for assertion keys.
  • MPI may choose any less restrictive (or equally restrictive) value for any info key.
  • MPI does not have to store user keys that it does not understand (it is not a cache).

Changes to the Text

TBD: Describe the text changes here.

Impact on Implementations

Almost useless but minimal work implementation: ignore all info keys, i.e. do not even store them. Always return INFO_NULL from all info query functions.

More useful but more work implementation: for each info key that can have some sort of internal effect on the MPI library, store the key as supplied by the user with the most appropriate value, i.e. the value that most closely represents what the MPI library will implement.

Impact on Users

This change addresses the current ambiguity regarding info query functions allowing the user to predict, and rely on, the results.

References

• Dec. 2016 - Plenary discussion: 12-2016 -- Info Query Part II.pdf

Problem specification slides were presented to the Sept 2016 meeting:
https://docs.google.com/presentation/d/10EPCx2LuJ0BWCeUXIG3pXJjqQs1tlnlO-6TJXIC8G8Q/edit?usp=sharing
The final slide was added during the face-to-face meeting to capture the result of discussion.

Insert any internal (other issues) or external (websites, papers, etc.) references here.

[markalle]

I think the "must not choose more restrictive values" is a little fuzzy. For example if comms could be created with "eager_buffer_size" "1000000" that might be a hint that the user knows will give smoother operation for their app, requiring less coordination and progression between their send/recv rank pairs. In this example there's no correctness requirement, it's just a hint that lets the app run faster if MPI can support it, so I wouldn't forbid MPI from using a smaller value.

I guess you could argue about the definition of the word "restrictive" and say the smaller value isn't more "restrictive" as there's no loss of functionality.

But in general are hints ever requests for functionality? I think they're the opposite: you give a hint to say you don't need some functionality, thus enabling MPI to possibly run faster due to not having to support whatever functionality your hint says you're not going to use. Eg "window_supports_lock_unlock" = "false" makes sense, the user is telling us they're not going to use lock/unlock perhaps allowing MPI to run faster in some way. There's no reason for a user to set "window_support_lock_unlock" = "true" you don't have to request functionality that's part of the standard.

I guess I can imagine hints that would be requests for functionality beyond what's in the standard. Like maybe "comm_supports_graceful_truncation" = "true" might allow MPI_Send(buf, 1000, MPI_CHAR, ...) and MPI_Recv(buf, 500, MPI_CHAR, ...) to be considered matching calls, where the recver would throw away the extra bytes. But it seems wrong somehow to require MPI to support this functionality beyond what's in the standard under the requirement that it must not choose a more restrictive value than was requested.

[dholmes-epcc-ed-ac-uk]

@markalle I agree. The INFO key mechanism for communicators is currently only used for hints, i.e. suggestions to the MPI library that do not restrict user behaviour at all. None of the existing key/value pairs defined by the MPI Standard (for communicators) is any more or less restrictive than any other.

We are planning to add new INFO keys that specify assertions (see #11 for the initial proposal, including the first 4 new assertion keys, and #58 for the bucket containing a whole bunch more suggested keys). For these new assertion keys, some values are more restrictive of user behaviour than others.

We will need language in the MPI Standard to cover both cases. For hints, all legal values are equally restrictive - so the MPI library has a free choice, just as it does now.

[dholmes-epcc-ed-ac-uk]

@markalle One reason for the user to set "key -> default value" or "key -> less restrictive than current value" would be to attempt to re-enable functionality that had previously been disabled. The MPI Standard could prohibit that attempt (that might be easier for us to specify). If the user is allowed to make the attempt (as currently proposed in this issue) then MPI must be allowed to ignore it.

MPI may wish to disallow such de-restriction. For example, it may be that a particular MPI library can implement a communicator more efficiently if, during communicator creation, the user asserts they will not use wildcards but it may not be possible to enable wildcards on that communicator without re-creating it.

[dholmes-epcc-ed-ac-uk]

@markalle we are envisioning INFO keys that can be assertions or hints - information from the user that might help MPI optimise its internal implementation or parameters. We have not considered using INFO to allow the user to request additional functionality from the MPI library. That would suggest a requested/provided mechanism like the thread support level interaction in MPI_INIT_THREAD.

[dholmes-epcc-ed-ac-uk]

An observation by @jsquyres "the functions are called _SET not _ADD or _MERGE" may considerably simplify things for us. The function name suggests it will replace the currently attached INFO object with the one supplied by the user.

We need a Venn diagram: old keys, new keys, same value keys, different value keys, can be supported key/values, cannot be supported key/values.

For new keys (not already set), MPI can choose to accept user-supplied key/value (if supported) or to ignore the key/value altogether.
For old keys (not present in user INFO), MPI can choose to remove the key/value (if supported) or to <what should go here? keep the old key/value? raise a 'not-supported' exception?>
For same value keys (already set and present with same value in user INFO), MPI must keep the key/value unchanged.
For different value keys (already set and present with different value in user INFO), MPI can choose to change the value to the user-supplied value (if supported) or to <what should go here? keep the old value? raise a 'not-supported' exception?>

[dholmes-epcc-ed-ac-uk]

@markalle for your particular example of a key that specifies a hint value for an internal parameter, e.g. "eager_buffer_size", the only reliable way to figure out if MPI has used your hint value or some different value for one of its internal parameters would be to use the Tools interface (MPI_T). If the MPI library has an eager buffer and if it exposes an MPI_T variable describing its size then you could query the actual setting of that MPI_T variable. That value might be different to the hint value in the INFO key, it might change over time without user intervention, it might not make any sense in a particular MPI implementation, etc.

[jsquyres]

@dholmes-epcc-ed-ac-uk and I discussed this on the phone yesterday. There are still large holes in this proposal, unfortunately.

In short, my $0.02: we are in the currently-bad situation that we are in because we have under-specified the behavior of these functions. And we are still trying to under-specify the behavior of these functions (albeit in different ways). This tactic is not working. We need to fully / completely specify what happens in cases including (but not limited to):

1. App sets no info values and does a GET. What exactly is returned?
2. App sets an info key/value that is used by the MPI implementation (e.g., it changes some internal behavior in the implementation), and then does a GET. What exactly is returned?
3. App sets an info key/value that is not used by the MPI implementation because the implementation does not understand the key (and ignores it), and then does a GET. What exactly is returned?
4. App sets an info key/value that is not used by the MPI implementation because the implementation just because it's Tuesday (even though the implementation does understand the key), and then does a GET. What exactly is returned?
5. App sets an info key/value that is not used by the MPI implementation because the value is non-sensical (e.g., the value is outside of a valid range), and then does a GET. What exactly is returned?
6. A time X, MPI implementation decides to change a value that was specified via SET at time (X-Y). Is this allowed?
   1. If so, how does the implementation differentiate between hints and assertions? (where assertions may have restrictions on what values they may asynchronously change to without affecting program correctness)
   2. Also, how exactly is this reported? Via a subsequent GET? Via MPI_T?
      • I argue that MPI_T is not a good solution here; MPI_T is supposed to be for tools -- it is explicitly not intended for end users / applications.
7. Is there a way to get back the keys/values that were SET vs. the keys/values that are currently being used? (other parts of MPI support this kind of functionality)

The question also occurs to me: if values returned by GET can change asynchronously from the app's point of view, how do we avoid race conditions between the values set by SET and returned by GET? More specifically: how can an app tell the difference between a) its value was rejected/ignored by SET, and b) its value was accepted/set by SET, but then later changed?