MPI Derived Datatypes, Scatter with ob1 and uct produces deadlocks

[AnnaLena77]

Background information

For certain research reasons, I need to use Open MPI with the pml ob1 in conjunction with Infiniband (ucx/uct as btl) on our cluster. This works largely without any problems.
In my particular program I am trying to split a matrix into sub-matrices (using Derived Datatypes) and distribute them to all processes using scatter.

What version of Open MPI are you using? (e.g., v4.1.6, v5.0.1, git branch name and hash, etc.)

Version 5.0.3

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

git clone

If you are building/installing from a git clone, please copy-n-paste the output from git submodule status.

+6f81bfd163f3275d2b0630974968c82759dd4439 3rd-party/openpmix (v1.1.3-3983-g6f81bfd1)
+4f27008906d96845e22df6502d6a9a29d98dec83 3rd-party/prrte (psrvr-v2.0.0rc1-4746-g4f27008906)
dfff67569fb72dbf8d73a1dcf74d091dad93f71b config/oac (heads/main)

Please describe the system on which you are running

• Operating system/version: NixOs
• Computer hardware: Cluster with AMD Opteron Processors 6272
• Network type: Infiniband (mlx4_0) / Ethernet

---

Details of the problem

The scatter always ends in a deadlock if the matrix is selected to be correspondingly large (here in the example 900x900, size 90 still works) and ob1 is used in conjunction with uct.
(I am using slurm scripts to distribute all jobs)

mpirun -n 81 --mca pml ucx ./test                  --> works
mpirun -n 81 --mca pml ob1 ./test                  --> deadlock
mpirun -n 81 --mca pml ob1 --mca btl ^uct ./test   --> works

If I use normal MPI data types (e.g. MPI_DOUBLE) instead of the Derived Datatypes, everything also works with uct. So the problem is definitely with the derived datatypes that are to be used.

MPI Program:

#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>
#include <math.h>
int main(int argc, char *argv[]) {
    MPI_Init(&argc, &argv);

    int rank, size;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);

    int N = 900;  //Matrix-Dimension
    int block_size = N / (int)sqrt(size); 

    int *matrix = NULL;

    if (rank == 0) {
        matrix = (int *)malloc(N * N * sizeof(int));
        for (int i = 0; i < N; i++) {
            for (int j = 0; j < N; j++) {
                matrix[i * N + j] = i * N + j;
            }
        }
    }
    MPI_Datatype blocktype, blocktype_resized;
    int array_of_sizes[2] = {N, N};
    int array_of_subsizes[2] = {block_size, block_size};
    int array_of_starts[2] = {0, 0};
    MPI_Type_create_subarray(2, array_of_sizes, array_of_subsizes, array_of_starts, MPI_ORDER_C, MPI_INT, &blocktype);
    MPI_Type_create_resized(blocktype, 0, block_size * sizeof(int), &blocktype_resized);
    MPI_Type_commit(&blocktype_resized);

    int *recvbuf = (int *)malloc(block_size * block_size * sizeof(int));
    
    MPI_Scatter(matrix, 1, blocktype_resized, recvbuf, block_size * block_size, MPI_INT, 0, MPI_COMM_WORLD);

    free(recvbuf);
    if (rank == 0) {
        free(matrix);
    }

    MPI_Type_free(&blocktype_resized);
    MPI_Finalize();
    return 0; 
}

[ggouaillardet]

There is a known issue in the coll/tuned component that can occur when using matching signatures but with different types and counts.

Instead of receiving block_size * block_size MPI_INT, what if you create a derived datatype made of block_size * block_size contiguous MPI_INT and then receive 1 instance of this newly created derived datatype?

[AnnaLena77]

Oh yes, that works fine, also with the btl uct.
Sorry, I've been searching for ages but haven't found the known issue.

[AnnaLena77]

I have to correct myself. Unfortunately, the approach does not work. I had first tried the whole thing with the same data type as receive-type (“blocktype_resize”). This no longer caused a deadlock, but this is not our goal and leads to problems in other situations. Integers of the size block_size*block_size must be received, not a blocktype_resize Datatype.

When we rebuilt our code today, we noticed this. A new derived datatype (created with MPI_Type_contiguous) still leads to a deadlock with uct.

MPI_Type_contiguous(block_size * block_size, MPI_INT, &recvtype);
MPI_Type_commit(&recvtype);
MPI_Scatter(matrix, 1, blocktype_resized, recvbuf, 1, recvtype, 0, MPI_COMM_WORLD);

--> Deadlock

[ggouaillardet]

I will try to reproduce the issue.

Meanwhile, you can try the following workaround and see how it goes

mpirun --mca coll ^tuned ...

[bosilca]

None of tuned scatter implementations have support for segmentation, thus scatter is immune to the datatype mismatch issue. I quickly scanned through the scatter algorithms in tuned, and as far as I see they all exchange the entire data in a single message (with different communication patterns, but the data goes in a single message. The UCT BTL is not up-to-date, if the tests runs with the TCP BTL then it is a clear indication where the problem is coming from.

[AnnaLena77]

I will try to reproduce the issue.

Meanwhile, you can try the following workaround and see how it goes

mpirun --mca coll ^tuned ...

Unfortunately, that doesn't work either. The deadlock remains at this point. The only time I no longer have a deadlock is when I use the tcp btl instead of uct:

mpirun --mca pml ob1 --mca btl ^uct  ./test

However, this is also not the best case for my workaround. Are there plans to fix the problem at some point?

[AnnaLena77]

I know the collective modules a little, as I have been working with the code for some time.
Tuned selects an algorithm from “base” based on criteria and executes it. HAN has the highest priority, but refers to tuned because the processes are not distributed evenly across the nodes.
If I lower the priority for both modules, the “base” module is still used automatically. Here in the example the Scatter Basic Linear Algorithm.
So even if I switch off "tuned", the same function (from base) is called in the collective module, so that the error occurs again in the uct btl.

[ggouaillardet]

@bosilca has a good point, and the issue is very unlikely the "known bug" I initially mentioned, but more likely something with btl/uct. Out of curiosity, why are you using pml/ob1 and btl/uct instead of pml/ucx?

In order to rule out the collective components, you can

mpirun --mca coll basic,libnbc,inter ...

and see if it helps.

[AnnaLena77]

I am working on an open MPI fork that measures internal performance data (similar to Peruse) at runtime and writes it to a database.

This performance data is then visualized in a frontend at runtime. It is designed for students to show them what is happening on the cluster at runtime.

Among other things, it is possible to visualize at runtime which algorithm is used for collective communication, how this affects the individual ranks, what they are doing at that moment, where potential late senders are, etc.

In ob1 pml I can use it to measure internal LateSender times, for example. I have been working with ob1 and its functionality for a long time and have learned to understand how it works. Our cluster has an Infiniband connection, so I use it in conjunction with uct. The runtimes with TCP are not acceptable.

UCX has a completely different architecture and I would have to start my work from scratch.

In addition, the students start their program from the frontend via button and of course have no knowledge of the various pmls, btls and collectives, so I have to execute an MPI command in the background that cannot be adapted due to the error.

Everything has worked wonderfully until the students started transferring derived datatypes via scatter.

[ggouaillardet]

I see, for the time being, you might want to give a try to Open MPI v4 and the btl/openib component.

[ggouaillardet]

@AnnaLena77 which UCX version are you using?

[AnnaLena77]

@AnnaLena77 which UCX version are you using?

1.16.0