MPI_Neighbor_allgather hang with reorderd Dist Graph

[wfdoran]

Thank you for taking the time to submit an issue!

Background information

In example on https://riptutorial.com/mpi/example/29195/graph-topology-creation-and-communication, the call
to MPI_Neighbor_allgather hangs using a dist graph with reorder. Note: if you set reorder to 0, works fine.

What version of Open MPI are you using? (e.g., v3.0.5, v4.0.2, git branch name and hash, etc.)

v4.1.1

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

/home/doran/OpenMPI> wget https://download.open-mpi.org/release/open-mpi/v4.1/openmpi-4.1.1.tar.gz
/home/doran/OpenMPI> gunzip openmpi-4.1.1.tar.gz
/home/doran/OpenMPI> tar -xf openmpi-4.1.1.tar
/home/doran/OpenMPI/openmpi-4.1.1> ./configure --prefix=/home/doran/OpenMPI/local
Open MPI configuration:
-----------------------
Version: 4.1.1
Build MPI C bindings: yes
Build MPI C++ bindings (deprecated): no
Build MPI Fortran bindings: mpif.h, use mpi, use mpi_f08
MPI Build Java bindings (experimental): no
Build Open SHMEM support: false (no spml)
Debug build: no
Platform file: (none)

Miscellaneous
-----------------------
CUDA support: no
HWLOC support: internal (external hlwoc version is less than internal version 2.0)
Libevent support: internal
PMIx support: Internal

Transports
-----------------------
Cisco usNIC: no
Cray uGNI (Gemini/Aries): no
Intel Omnipath (PSM2): no
Intel TrueScale (PSM): no
Mellanox MXM: no
Open UCX: no
OpenFabrics OFI Libfabric: no
OpenFabrics Verbs: yes
Portals4: no
Shared memory/copy in+copy out: yes
Shared memory/Linux CMA: yes
Shared memory/Linux KNEM: no
Shared memory/XPMEM: no
TCP: yes

Resource Managers
-----------------------
Cray Alps: no
Grid Engine: no
LSF: no
Moab: no
Slurm: yes
ssh/rsh: yes
Torque: no

OMPIO File Systems
-----------------------
DDN Infinite Memory Engine: no
Generic Unix FS: yes
IBM Spectrum Scale/GPFS: no
Lustre: no
PVFS2/OrangeFS: no

/home/doran/OpenMPI/openmpi-4.1.1> make
/home/doran/OpenMPI/openmpi-4.1.1> make install

If you are building/installing from a git clone, please copy-n-paste the output from git submodule status.

Please describe the system on which you are running

• Operating system/version:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.5 LTS
Release:        18.04
Codename:       bionic

running inside WSL 2.

Issue also occurs on RedHat 7 desktop.

• Computer hardware:

Intel Core i7-9700 @ 3.00GHz
Coffee Lake
8 Cores / 8 Threads
16 GB Ram
Dell OT2HR0 main board

• Network type:

None. Desktop

---

Details of the problem

Please describe, in detail, the problem that you are having, including the behavior you expect to see, the actual behavior that you are seeing, steps to reproduce the problem, etc. It is most helpful if you can attach a small program that a developer can use to reproduce your problem.

Note: If you include verbatim output (or a code block), please use a GitHub Markdown code block like below:

$ cat example.c
// https://riptutorial.com/mpi/example/29195/graph-topology-creation-and-communication

#include <mpi.h>
#include <stdio.h>

#define nnode 4

int main()
{
    MPI_Init(NULL, NULL);

    int rank;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);

    int source = rank;
    int degree;
    int dest[nnode];
    int weight[nnode] = {1, 1, 1, 1};
    int recv[nnode] = {-1, -1, -1, -1};
    int send = rank;

    // set dest and degree.
    if (rank == 0)
    {
        dest[0] = 1;
        dest[1] = 3;
        degree = 2;
    }
    else if(rank == 1)
    {
        dest[0] = 0;
        degree = 1;
    }
    else if(rank == 2)
    {
        dest[0] = 3;
        dest[1] = 0;
        dest[2] = 1;
        degree = 3;
    }
    else if(rank == 3)
    {
        dest[0] = 0;
        dest[1] = 2;
        dest[2] = 1;
        degree = 3;
    }

    // create graph.
    MPI_Comm graph;
    int reorder = 1;
    MPI_Dist_graph_create(MPI_COMM_WORLD, 1, &source, &degree, dest, weight, MPI_INFO_NULL, reorder, &graph);

    // send and gather rank to/from neighbors.
    printf("AAA %d\n", rank); fflush(stdout);
    MPI_Neighbor_allgather(&send, 1, MPI_INT, recv, 1, MPI_INT, graph);
    printf("BBB %d\n", rank); fflush(stdout);

    printf("Rank: %i, recv[0] = %i, recv[1] = %i, recv[2] = %i, recv[3] = %i\n", rank, recv[0], recv[1], recv[2], recv[3]);

    MPI_Finalize();
    return 0;
}

$ mpicc example.c -o example
$ mpirun -np 4 ./example
AAA 0
BBB 0
Rank: 0, recv[0] = 3, recv[1] = 0, recv[2] = 2, recv[3] = -1
AAA 1
BBB 1
Rank: 1, recv[0] = 1, recv[1] = 3, recv[2] = 2, recv[3] = -1
AAA 2
BBB 2
Rank: 2, recv[0] = 2, recv[1] = -1, recv[2] = -1, recv[3] = -1
AAA 3
^C^C

Expected output with reorder=1

$ mpicc example.c -o example
$ mpirun -np 4 ./example
AAA 0
BBB 0
Rank: 0, recv[0] = 2, recv[1] = 3, recv[2] = 1, recv[3] = -1
AAA 1
BBB 1
Rank: 1, recv[0] = 2, recv[1] = 3, recv[2] = 0, recv[3] = -1
AAA 2
BBB 2
Rank: 2, recv[0] = 3, recv[1] = -1, recv[2] = -1, recv[3] = -1
AAA 3
BBB 3
Rank: 3, recv[0] = 2, recv[1] = 0, recv[2] = -1, recv[3] = -1

I did a bit of printf debugging. With reorder set to 1, I tracked it to

mca_coll_basic_neighbor_allgather_dist_graph()

in ompi/mca/coll/basic/coll_basic_neighbor_allgather.c

All four ranks get to and enter
rc = ompi_request_wait_all (indegree + outdegree, reqs, MPI_STATUSES_IGNORE);
but rank 3 does not return from this. Because ompi_request_wait_all is #defined to
be a function pointer, I could not immediately track it any further and decided to
submit this issue report.

[cedhub]

Any news on this topic ? Just realised that my program did not work because of this reorder flag set to 1.
Is this fixed in v5.x.x ?

I'm running v4.1.3 on macOS 12.5, Apple Clang version 13.1.6