What features do users need from an MPI C++ interface?

[jeffhammond]

This is a meta-issue, which I am creating to capture user feedback on MPI C++ bindings.

I am moving this over from https://scicomp.stackexchange.com/questions/7978/what-features-do-users-need-from-an-mpi-c-interface, which was extremely well-received despite not complying with the rules of StackExchange.

Original Prompt

The 3.0 version of the MPI standard formally deleted the C++ interface (it was previously deprecated). While implementations may still support it, features that are new in MPI-3 do not have a C++ interface defined in the MPI standard. See http://blogs.cisco.com/performance/the-mpi-c-bindings-what-happened-and-why/ for more information.

The motivation for removing the C++ interface from MPI was that it had no significant value over the C interface. There were very few differences other than "s/_/::/g" and many features that C++ users are accustomed to were not employed (e.g. automatic type determination via templates).

As someone who participates in the MPI Forum and works with a number of C++ projects that have implemented their own C++ interface to the MPI C functions, I would like to know what are the desirable features of a C++ interface to MPI. While I commit to nothing, I would be interested in seeing the implementation of a standalone MPI C++ interface that meets the needs of many users.

And yes, I am familiar with Boost::MPI but it only supports MPI-1 features and the serialization model would be extremely difficult to support for RMA.

One C++ interface to MPI that I like is that of Elemental's mpi wrapper so perhaps people can provide some pro and con w.r.t. that approach. In particular, I think MpiMap solves an essential problem.

[jeffhammond]

Wolfgang Bangerth provided the following response (https://scicomp.stackexchange.com/a/7991/150):

Let me first answer why I think C++ interfaces to MPI have generally not been overly successful, having thought about the issue for a good long time when trying to decide whether we should just use the standard C bindings of MPI or building on something at higher level:

When you look at real-world MPI codes (say, PETSc, or in my case deal.II), one finds that maybe surprisingly, the number of MPI calls isn't actually very large. For example, in the 500k lines of deal.II, there are only ~100 MPI calls. A consequence of this is that the pain involved in using lower-level interfaces such as the MPI C bindings, is not too large. Conversely, one would not gain all that much by using higher level interfaces.

My second observation is that many systems have multiple MPI libraries installed (different MPI implementations, or different versions). This poses a significant difficulty if you wanted to use, say, boost::mpi that don't just consist of header files: either there needs to be multiple installations of this package as well, or one needs to build it as part of the project that uses boost::mpi (but that's a problem in itself again, given that boost uses its own build system, which is unlike anything else).

So I think all of this has conspired against the current crop of C++ interfaces to MPI: The old MPI C++ bindings didn't offer any advantage, and external packages had difficulties with the real world.

This all said, here's what I think would be the killer features I would like to have from a higher-level interface:

• It should be generic. Having to specify the data type of a variable is decidedly not C++-like. Of course, it also leads to errors. Elemental's MpiMap class would already be a nice first step (though I can't figure out why the heck the MpiMap::type variable isn't static const, so that it can be accessed without creating an object).

• It should have facilities for streaming arbitrary data types.

• Operations that require an MPI_Op argument (e.g., reductions) should integrate nicely with C++'s std::function interface, so that it's easy to just pass a function pointer (or a lambda!) rather than having to clumsily register something.

boost::mpi actually satisfies all of these. I think if it were a header-only library, it'd be a lot more popular in practice. It would also help if it supported post-MPI 1.0 functions, but let's be honest: this covers most of what we need most of the time.

[jeffhammond]

@gnzlbg provided the following response (https://scicomp.stackexchange.com/a/14640/150):

My list in no particular order of preference. The interface should:

• be header only, without any dependencies but <mpi.h>, and the standard library,
• be generic and extensible,
• be non-blocking only (if you want to block, then block explicitly, not by default),
• allow continuation-based chaining of non-blocking operations,
• support extensible and efficient serialization (Boost.Fusion like, such that it works with RMA),
• have zero abstraction penalty (i.e. be at least as fast as the C interface),
• be safe (the destructor of a non-ready future is called? -> std::terminate!),
• have a strong DEBUG mode with tons of assertions,
• extremely type-safe (no more ints/void* for everything, heck I want tags to be types!),
• it should work with lambdas (e.g. all reduce + lambda),
• use exceptions consistently as error-reporting and error-handling mechanism (no more error codes! no more function output arguments!),
• MPI-IO should offer a non-blocking I/O interface in the style of Boost.AFIO,
• and just follow good modern C++ interface design practices (define regular types, non-member non-friend functions, play well with move semantics, support range operations, ...)

Extras:

• allow me to chose the executor of the MPI environment, that is, which thread pool it uses. Right now you can have applications with a mix of OpenMP, MPI, CUDA, and TBB... all at the same time, where each run-time thinks it owns the environment and thus ask the operating system for threads every time they feel like it. Seriously?

• use the STL (and Boost) naming convention. Why? Every C++ programmer knows it.

I want to write code like this:

    auto buffer = some_t{no_ranks};
    auto future = gather(comm, root(comm), my_offsets, buffer)
                  .then([&](){
                    /* when the gather is finished, this lambda will 
                       execute at the root node, and perform an expensive operation
                       there asynchronously (compute data required for load 
                       redistribution) whose result is broadcasted to the rest 
                       of the communicator */
                    return broadcast(comm, root(comm), buffer);
                  }).then([&]() {
                    /* when broadcast is finished, this lambda executes 
                       on all processes in the communicator, performing an expensive
                       operation asynchronously (redistribute the load, 
                       maybe using non-blocking point-to-point communication) */
                     return do_something_with(buffer);
                  }).then([&](auto result) {
                     /* finally perform a reduction on the result to check
                        everything went fine */
                     return all_reduce(comm, root(comm), result, 
                                      [](auto acc, auto v) { return acc && v; }); 
                  }).then([&](auto result) {
                      /* check the result at every process */
                      if (result) { return; /* we are done */ }
                      else {
                        root_only([](){ write_some_error_log(); });
                        throw some_exception;
                      }
                  });

    /* Here nothing has happened yet! */
 
    /* ... lots and lots of unrelated code that can execute concurrently 
       and overlaps with communication ... */

    /* When we now call future.get() we will block 
       on the whole chain (which might have finished by then!).
    */
    
    future.get();

Think how one could chain all this operations using MPI_C's requests. You would have to test at multiple (or every single) intermediate step through a whole lot of unrelated code to see if you can advance your chain without blocking.

[jeffhammond]

GradGuy provided the following response (https://scicomp.stackexchange.com/a/8009/150):

Personally, I don't really mind calling long C-style functions for the exact reason Wolfgang mentioned; there are really few places you need to call them and even then, they almost always get wrapped around by some higher-level code.

The only things that really bother me with C-style MPI are custom datatypes and, to a lesser degree, custom operations (because I use them less often). As for custom datatypes, I'd say that a good C++ interface should be able to support generic and efficient way of handling this, most probably through serialization. This is of course the route that boost.mpi has taken, which if you are careful, is a big time saver.

As for boost.mpi having extra dependencies (particularly boost.serialization which itself is not header-only), I've recently came across a header-only C++ serialization library called cereal which seems promising; granted it requires a C++11 compliant compiler. It might worth looking into and using it as a based for something similar to boost.mpi.

[jeffhammond]

Utkarsh Bhardwaj provided the following response (https://scicomp.stackexchange.com/a/25094/150):

The github project easyLambda provides a high level interface to MPI with C++14.

I think the project has similar goals and it will give some idea on things that can be and are being done in this area by using modern C++. Guiding other efforts as well as easyLambda itself.

The initial benchmarks on performance and lines of code have shown promising results.

Following is a short description of features and interface it provides.

The interface is based on data flow programming and functional list operations that provide inherent parallelism. The parallelism is expressed as property of a task. The process allocation and data distribution for the task can be requested with a .prll() property. There are good number of examples in the webpage and code-repository that include LAMMPS molecular dynamics post processing, explicit finite difference solution to heat equation, logistic regression etc. As an example the heat diffusion problem discussed in the article HPC is dying... can be expressed in ~20 lines of code.

I hope it is fine to give links rather than adding more details and example codes here.

Disclamer: I am the author of the library. I believe I am not doing any harm in hoping to get a constructive feedback on the current interface of easyLambda that might be advantageous to easyLambda and any other project that pursues similar goals.

[mhoemmen]

Given how fast the C++ Standard is moving with respect to thread and task parallelism, coroutines, networking, and reflection, it seems premature to standardize a C++ MPI interface now. Why not let all these great libraries first build experience presenting a modern C++ interface to the latest MPI features? Why repeat the mistake of the '90s and rush to standardize? I would love for someone to modernize Boost.MPI, for example; I would be happy to help with that (at least to test changes).

If we want gather(...).then(...).then(...)...., then why not build on the C++ networking TS? If we worry about thread interactions, then why not wait on (or participate in) an executors-networking merger? I can guess some reasons why, but I would expect an MPI proposal to answer questions like that.

Regarding a header-only library: this sounds good if you're starting a new project, but some existing C++ projects that use MPI care a lot about build sizes and times. If we want to put something in the MPI Standard, I'd like to see some build experiments in real applications.

[mhoemmen]

Wolfgang Bangerth wrote:

My second observation is that many systems have multiple MPI libraries installed (different MPI implementations, or different versions). This poses a significant difficulty if you wanted to use, say, boost::mpi that don't just consist of header files: either there needs to be multiple installations of this package as well, or one needs to build it as part of the project that uses boost::mpi (but that's a problem in itself again, given that boost uses its own build system, which is unlike anything else).

We've dealt with this issue of multiple MPI installations by writing an MPI (C binding) library that just calls through to an underlying MPI implementation. Our library dispatches to an underlying MPI implementation at run time via dlopen or the Windows equivalent (it works great on Windows). We don't expose any details of the underlying MPI implementation's ABI, so it's handy for things like Python bindings. Our library takes effort to maintain and incurs function call overhead, but it's been useful enough that we're thinking about open-sourcing it. If you're interested, please let me know.

[omor1]

We've dealt with this issue of multiple MPI installations by writing an MPI (C binding) library that just calls through to an underlying MPI implementation. Our library dispatches to an underlying MPI implementation at run time via dlopen or the Windows equivalent (it works great on Windows). We don't expose any details of the underlying MPI implementation's ABI, so it's handy for things like Python bindings.

Unrelated to the discussion at hand, but I'm curious as to how do you deal with the opaque handles (e.g. MPI_Comm, MPI_Request) that are exposed via mpi.h? These are highly implementation-dependent features whose sizes do depend on the underlying ABI. There was discussion of exactly this issue in #159. As a concrete example: in Open MPI, handles are pointers, while in MPICH-derivatives, they are int.

[omor1]

Regarding a header-only library: this sounds good if you're starting a new project, but some existing C++ projects that use MPI care a lot about build sizes and times. If we want to put something in the MPI Standard, I'd like to see some build experiments in real applications.

There are both benefits and detriments to defining the MPI C++ interface so that it can be implemented as a header-only library. An obvious benefit is that a single generic implementation may be sufficient for all underlying MPI libraries, which can ease adoption and maintenance burden. The flip side is that then there are severe restrictions on the e.g. datatypes interface, as they would be required to use the MPI C interface rather than whatever low-level representation is used by the implementation.

[mhoemmen]

@acdemiralp wrote:

Why not co-develop it along with the C++ standard?

Yes -- let's write a library first, then standardize it. Maybe that means becoming a Boost.MPI developer or taking over Boost.MPI development, or maybe it means starting a new library (if one can make a strong technical argument that Boost.MPI has a fundamentally flawed design).

[sg0]

Thanks for initiating the discussion, Jeff. I am unsure if a number of ubiquitous C++ idioms can be supported by an MPI C++ binding (for e.g., RAII, because a C++ destructor can be called after MPI_Finalize). As such, perhaps we can identify the C++ idioms that can be supported in a conformant way in such a binding, since in C++ there are potentially different ways to implement/design an interface. In terms of ownership, since MPI does not own the data and request buffers (users responsibility), the C++ interface must follow suit. However, from the example mentioned by Mark H., it seems the return object of the MPI function invocation is a future. From my discussion with a few other forum members, it seems future objects can represent MPI request objects; that means the MPI C++ interface have to maintain the intermediate futures. Futures may require ownership transfer in certain cases, which involve extra copies. It seems for a C++ user, allowing an interface that accepts C++20 ranges[*] could be quite useful (not using ranges from std:: but implementing it keeping the interface). But, this would require 'hiding' (hence maintaining) derived datatypes, so again I don't know if passing this responsibility to the C++ API is appropriate performance-wise (may require extra copies during scope transitions). [*] https://en.cppreference.com/w/cpp/ranges <https://en.cppreference.com/w/cpp/ranges> A templated C++ free-function based approach will perhaps be the easiest to implement and lead to the least overhead. But, that means we won't be making use of the modern C++ functionalities.

…

On Mon, Apr 27, 2020 at 7:44 PM Mark Hoemmen ***@***.***> wrote: @acdemiralp <https://github.com/acdemiralp> wrote: Why not co-develop it along with the C++ standard? Yes -- let's write a library first, then standardize it. Maybe that means becoming a Boost.MPI developer or taking over Boost.MPI development, or maybe it means starting a new library (if one can make a strong technical argument that Boost.MPI has a fundamentally flawed design). — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#288 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCNC6IRKDBSJIIE5643S43ROY7KBANCNFSM4MQE2IGA> .

-- Sayan | https://sg0.github.io/

[mhoemmen]

@sg0 wrote:

However, from the example mentioned by Mark H., it seems the return object of the MPI function invocation is a future.

It would be a sender, in P0443R13 terms, not a future. Senders and receivers avoid some of the shared state issues that futures have.

In any case, I'm not necessarily advocating this design. I'm just saying that if people want that kind of design, then it should fit with how modern C++ is doing it. I'd like to see the people doing that design engage with C++ networking and executors experts.

[StellarTodd]

We've dealt with this issue of multiple MPI installations by writing an MPI (C binding) library that just calls through to an underlying MPI implementation. Our library dispatches to an underlying MPI implementation at run time via dlopen or the Windows equivalent (it works great on Windows). We don't expose any details of the underlying MPI implementation's ABI, so it's handy for things like Python bindings.

Unrelated to the discussion at hand, but I'm curious as to how do you deal with the opaque handles (e.g. MPI_Comm, MPI_Request) that are exposed via mpi.h? These are highly implementation-dependent features whose sizes do depend on the underlying ABI. There was discussion of exactly this issue in #159. As a concrete example: in Open MPI, handles are pointers, while in MPICH-derivatives, they are int.

We defined a Handle class that contains a union, and conversions methods for converting back and forth between native handles and our handles. The conversions are done in the plugin portion of the library that is compiled against a specific MPI implementation.

Since this is off topic, I don't want to get into any more details here. Feel free to contact Mark or me for further details.

[jeffhammond]

Given how fast the C++ Standard is moving with respect to thread and task parallelism, coroutines, networking, and reflection, it seems premature to standardize a C++ MPI interface now. Why not let all these great libraries first build experience presenting a modern C++ interface to the latest MPI features? Why repeat the mistake of the '90s and rush to standardize? I would love for someone to modernize Boost.MPI, for example; I would be happy to help with that (at least to test changes). As a person who also considered updating Boost.MPI, and then walked through a 900+ page standard to see if it is feasible to write a full C++17 wrapper around it from scratch, and then gave up on all these due to the solo amount involved and used barebones C for MPI in an otherwise fully modern C++17 application: Why not co-develop it along with the C++ standard? The majority of the features you mention are already concrete, and even provide experimental/predecessor implementations.

You don’t have to implement everything to make an impact on the MPI Forum. If you look at the BigMPI stuff I did, I hit most or all of the relevant functions but didn’t support datatypes. People understood how to generalize. As for Boost.MPI enhancements, adding support for nonblocking collectives, Mprobe/Mrecv, and neighborhood collectives is both important and straightforward. RMA will be hard but just leave that for now. It doesn’t make it or break it for either goal. In any case, if you are serious about Boost.MPI3, setup a repo for it, add the classes of functionality you want to support, and start with easy stuff like Mrecv. Tag me in any issues where you need help understanding the document. It has been a while but I have read it cover to cover at least once, and the meaty stuff many times. You might also look at code generation methods like mpiwrap from LLNL to understand how to automate away some of the tedium. It’s not designed for this purpose, but it might be useful anyways.

[raffenet]

FYI https://gitlab.com/correaa/boost-mpi3. I don't know any of the details of the implementation, just that it exists and some projects have investigated using it.