ulfm: implement ulfm that does not depend on PMI

[hzhou]

Pull Request Description

The current ULFM implementation is based on PMI/hydra, in particular, the PMI_dead_processes from PMI_Get. This limits the usage on systems that use an unsupported PMI.

In this new implementation, we start with a resilient implementation of MPIX_Comm_agree, which relies on using active message to probe processes in a binary tree fashion. We rely on the fact that an am_send to a dead process will result in a immediate failure.

This resilient MPIX_Comm_agree can provide an consensus on MPIX_Comm_get_failed and a simple re-implementation of MPIX_Comm_shrink.

[skip warnings]

Test Program

if (rank == 1) {
    exit(-1);
}
int flag = 0x1;
MPIX_Comm_agree(comm, &flag);

MPI_Group failed_group; int num_failed;
MPIX_Comm_get_failed(comm, &failed_group);
MPI_Group_size(failed_group, &num_failed);
if (num_failed > 0) {
    MPI_Comm new_comm;
    MPIX_Comm_shrink(comm, &new_comm);
    /* error checking */
    MPI_Comm_free(&comm);
    comm = new_comm;
}

Notes

• We rely on a send_probe function that will fail if the target process is dead. In ch4, this relies on MPIDI_NM_am_isend. However -
  • it may fail right away by returning an error
  • it may succeed then set sreq->status.MPI_ERROR upon completion
  • it may succeed all the way but the target process dies after the probe
  • it may succeed with no feedback -- consider injection send
  • it may infinitely give EAGAIN in progress

Author Checklist

• Provide Description
  Particularly focus on why, not what. Reference background, issues, test failures, xfail entries, etc.
• Commits Follow Good Practice
  Commits are self-contained and do not do two things at once.
  Commit message is of the form: module: short description
  Commit message explains what's in the commit.
• Passes All Tests
  Whitespace checker. Warnings test. Additional tests via comments.
• Contribution Agreement
  For non-Argonne authors, check contribution agreement.
  If necessary, request an explicit comment from your companies PR approval manager.