hydra from mpich 5.0.0 does not fall back to PMI 1.1

[garlick]

Problem: HPE noted that the Flux instructions for launching Flux framework with hydra no longer worked for MPICH 5 here:

• Broker immediately exits when starting Flux with Hydra flux-framework/flux-core#7419

Flux (client) sends the PMI version = 1.1. Hydra (server) responds with version = 1.2. Flux throws an error.

It seems like the server should respond with 1.1, if that is possible. In other words, the version negotiation should determine the highest version that both client and server support.

We'll get PMI 1.2 implemented in flux but until then, fallback would be handy.

• libpmi: support MPICH's PMI 1.2 flux-framework/flux-core#7420

[hzhou]

PMI 1.2 is fully backward compatible so a PMI 1.1 client should be able to work with a PMI1.2 server without change. The reverse won't work since PMI 1.1 server won't be able to respond to PMI 1.2 client commands. The idea to have server reply its highest supported version is to let client discover server capabilities so it can choose feature paths.

I understand the intuition for server to reply 1.1 if the client requested 1.1. What should the behavior be if client request 1.2 and server is old and does not aware of 1.2? Since the PMI client fully controls what API functions to use and plus the backward compatibility condition within major versions, the current design is to let server always reply its highest supported version (within the major version) and let the client decide whether to proceed or not.

[garlick]

Ah thanks, I had assumed that wasn't intentional. OK!

I think the PMI 2 version negotiation is compatible with PMI 1, but its wire protocol isn't, so presumably this rule would just apply to minor protocol versions?

(Edit: we have both PMI client and server in Flux so I want to be sure I get it right!)

[hzhou]

The incompatible wire protocol in PMI2 is a problem! So the client need pick a major version and the server will reply within the major version. Although we still maintain both PMI 2 in both hydra and mpich, we have deprecated PMI 2. All third party that supports PMI 2 also support PMI 1 to our knowledge. All future development will be focused in PMI 1 with backward-compatible minor versions. In the future if there's development that need break the backward compatibility, I guess it will be PMI 3 :)

[garlick]

I'm not sure if its too late to change the meaning of the protocol negotiation at this point, but if the server responded with the MIN(client_version, server_version), doesn't that work in all cases, and is intuitive for both server and client implementors?

What should the behavior be if client request 1.2 and server is old and does not aware of 1.2

Server responds with 1.1 and client dynamically disables features that require 1.2 like groups or emulates them somehow?

[hzhou]

I'm not sure if its too late to change the meaning of the protocol negotiation at this point, but if the server responded with the MIN(client_version, server_version), doesn't that work in all cases, and is intuitive for both server and client implementors?

That will work if the server does not error out. In the spirit of backward compatibility, we try to let new MPICH client be able to work with old Cray PE or Slurm. Currently that means we need advertise the minimum protocol we need to the server. It is easy to convince the user to upgrade their MPI library than to convince a system to upgrade their PE.

[hzhou]

I guess that is an opposite situation as Flux need to deal with ;)

[garlick]

We have to deal with both - we provide PMI to MPI (server) and we use PMI to bootstrap Flux (client). So you could say we have all the problems, like you since you have both hydra and mpi :-)

I see two problems with the way you are proposing

• It implies that servers have to implement all prior versions to their max supported version
• if wire protocol ever changes in a future version, the client and server won't have agreed upon a wire protocol

The only implementation of the wire protocol in slurm is in the --mpi=pmi2 plugin, which is a 2.0 implementation that falls back to 1.1. We use this a lot to launch flux in a slurm allocation. This is how it currently works (and has for years):

server drops down to whatever client sends (if supported):
https://github.com/SchedMD/slurm/blob/master/src/plugins/mpi/pmi2/agent.c#L242-L287

set_pmi_version() only supports 1.1 and 2.0:
https://github.com/SchedMD/slurm/blob/master/src/plugins/mpi/pmi2/client.c#L83-L107

So we are screwed either way by Slurm. If flux client implements 1.2 and sends minimum as 1.1, server will always use 1.1 even if slurm implements 1.2. If flux client sends 1.2 as client maximum, server will just abort.

[garlick]

if wire protocol ever changes in a future version, the client and server won't have agreed upon a wire protocol

That was probably confusing. I mean the low level framing like the difference between PMI 1 and 2

[hzhou]

I see two problems with the way you are proposing

* It implies that servers have to implement all prior versions to their max supported version

Yes. And that's what backward compatibility means. The server need ensure all prior versions still work within a major version.

* if wire protocol ever changes in a future version, the client and server won't have agreed upon a wire protocol

I think currently the wire protocol is bound to the major version.

The only implementation of the wire protocol in slurm is in the --mpi=pmi2 plugin, which is a 2.0 implementation that falls back to 1.1. We use this a lot to launch flux in a slurm allocation. This is how it currently works (and has for years):

It is weird to me that Slurm call it pmi2, but in reality it supports PMI1.1 as well as PMI 2.0.

So we are screwed either way by Slurm. If flux client implements 1.2 and sends minimum as 1.1, server will always use 1.1 even if slurm implements 1.2. If flux client sends 1.2 as client maximum, server will just abort.

What's wrong with server using 1.1? I assume by the time Slurm implements 1.2, they need think about what is the correct server response as we are considering now. Dropping down to 1.1 all it means is some new application that is using the new MPI session features does not work, but for the majority of MPI applications, 1.1 should work fine.

[hzhou]

@garlick Out of curiosity, what is Flux's PMIx strategy?

[garlick]

We have a flux pmix plugin here: https://github.com/flux-framework/flux-pmix that does the bare minimum to bootstrap openpmi 5. That's as far as I was willing to take it. Since the spec is absolutely huge, I figure others can pick and choose what they are interested in if they want to build it up from there.

I like the direction you are going with PMI 1.x. The design of PMI was pretty smart IMHO and what was the best about it was the separation of concerns between parallel program and launcher. That gets a bit muddy in the real world, but PMIx just completely gives up on the abstraction.

[garlick]

I think currently the wire protocol is bound to the major version.

Yes that's what I mean. If I send the lowest version I can support, 1.1 and server sends the highest version it can support 4.1, then I say "cool, I can use pmi 3.1" and start sending pmi 3 message frames. From server perspective, the version handshake doesn't tell it which kind of frame is coming, 1, 2, 3, or 4.

[hzhou]

I think currently the wire protocol is bound to the major version.

Yes that's what I mean. If I send the lowest version I can support, 1.1 and server sends the highest version it can support 4.1, then I say "cool, I can use pmi 3.1" and start sending pmi 3 message frames. From server perspective, the version handshake doesn't tell it which kind of frame is coming, 1, 2, 3, or 4.

We can strive for that with the exception for PMI 2. I really want to drop PMI 2 support at some point.

[garlick]

DO you agree the lack of agreement on a major version (assuming that affects framing) is an issue then?
If so, it seems like the "send max version, respond with max commonly supported version" is the right way to do that?

If 2.x were deprecated, client could send 2.0, server could respond with 1.2. IOW server can skip implementing any version lke 2.0 and still implement later versions.

[hzhou]

I can agree to that eventually. Meanwhile, I need make sure if MPICH is build by requesting PMI1.2, it won't fail on systems running Slurm or Cray -- I am assuming it will fail. It is worthwhile to survey and find out.