TCP BTL fails to collect all interface addresses (when interfaces are on different subnets)

[JBlaschke]

This issue is related to: #5818

I am encountering

Open MPI detected an inbound MPI TCP connection request from a peer
that appears to be part of this MPI job (i.e., it identified itself as
part of this Open MPI job), but it is from an IP address that is
unexpected.  This is highly unusual.

The inbound connection has been dropped, and the peer should simply
try again with a different IP interface (i.e., the job should
hopefully be able to continue).

  Local host:          nid002292
  Local PID:           1273838
  Peer hostname:       nid002293 ([[9279,0],1])
  Source IP of socket: 10.249.13.210
  Known IPs of peer:
	10.100.20.22
	128.55.69.127
	10.249.13.209
	10.249.36.5
	10.249.34.5

when running the CP2K container (https://catalog.ngc.nvidia.com/orgs/hpc/containers/cp2k) on NERSC's Perlmutter system(https://docs.nersc.gov/systems/perlmutter/architecture/). We've tried OpenMPI v4.1.2rc2 and v4.1.5

Background

Perlmutter's GPU nodes have 4 NICS, each has a private IP address, and one NIC (the one corresponding to the hsn0 interface has an additional public IP addres -- therefore each node has one NIC with two addresses, and these addresses are in different subnets. Eg.:

blaschke@nid200257:~> ip -4 -f inet addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: nmn0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    altname enp195s0
    inet 10.100.108.32/22 brd 10.100.111.255 scope global nmn0
       valid_lft forever preferred_lft forever
3: hsn0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 10000
    altname enp194s0
    inet 10.249.41.248/16 brd 10.249.255.255 scope global hsn0
       valid_lft forever preferred_lft forever
    inet 128.55.84.128/19 brd 128.55.95.255 scope global hsn0:chn
       valid_lft forever preferred_lft forever
4: hsn1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 10000
    altname enp129s0
    inet 10.249.41.232/16 brd 10.249.255.255 scope global hsn1
       valid_lft forever preferred_lft forever
5: hsn2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 10000
    altname enp66s0
    inet 10.249.41.231/16 brd 10.249.255.255 scope global hsn2
       valid_lft forever preferred_lft forever
6: hsn3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 10000
    altname enp1s0
    altname ens3
    inet 10.249.41.247/16 brd 10.249.255.255 scope global hsn3
       valid_lft forever preferred_lft forever

(the example above also shows the node management network nmn interace -- but MPIshouldn't be talking to that anyway)

I think the error must be caused by hsn0's two ip addresses on two subnets.

[jsquyres]

I'm a little confused about what you are trying to do.

Your description of the problem lists 6 private IP addresses and 1 public IP address:

1. lo: 127.0.0.1/8
2. nm0: 10.100.108.32/22
3. hsn0: 10.249.41.248/16 and 128.55.85.128/19
4. hsn1: 10.249.41.232/16
5. hsn2: 10.249.41.231/16
6. hsn3: 10.249.41.247/16

A few questions:

1. What interface / IP address do you want Open MPI to use?
2. Why are all the hsnX interfaces on the same 10.249.x.y/16 subnet? That is... confusing, and may not be what you really want.
3. How does the container networking come into play here? I.e., what are the interfaces and IP addresses of the container(s) involved? Overlay networks add another whole level of complexity if that networking is different than the host networking.

[JBlaschke]

@jsquyres thanks for getting back to me

1. What interface / IP address do you want Open MPI to use?

I am not familiar with the logic that OpenMPI's TCP BTL uses to bind to networks interfaces. Since each node has one NIC/GPU and NUMA node, the ideal case would be for OpenMPI to use the "nearest" NIC with respect to the rank's NUMA domain. That implies an optimal choice of interface for each rank. It doesn't matter what address it uses.

2. Why are all the hsnX interfaces on the same 10.249.x.y/16 subnet? That is... confusing, and may not be what you really want.

I don't see why we would want 4 subnets. We have 4 NICS per node (and therefore 4 hsnX interfaces) because there are 4 GPUs per node. This way we can do things like GPU-Direct MPI traffic between GPUs on different nodes without traffic from different GPUs interfering. This is not helped by NICs being on separate subnets.

More broadly speaking: You can think of Perlmutter's high-speed network to be a single private network, and we've given each node a "line to the outside world" by piggy-backing off the hsn0's NIC by giving it a public IP.

Based on the error message:

  Local host:          nid002292
  Local PID:           1273838
  Peer hostname:       nid002293 ([[9279,0],1])
  Source IP of socket: 10.249.13.210
  Known IPs of peer:
	10.100.20.22
	128.55.69.127
	10.249.13.209
	10.249.36.5
	10.249.34.5

I think what's happening is that OpenMPI is unaware that 10.249.13.210 is also a valid IP address for nid002293. This is speculation now: the fact that I see a 128.55.X.X address in this list, (and not a 10.249.X.X one in its place) is that OpenMPI recorded the 128.55.X.X address as "the" address of hsn0. Yet whichever rank on nid002293 that tried to initiate the connection must have used the 10.249.X.X address. I don't know the logic that OpenMPI uses, but maybe it's prioritizing different networks when collecting valid IPs compared to when it is initiating connections.

Perhaps the best solution is to record all IPs (even if they belong to different subnets)?

3. How does the container networking come into play here? I.e., what are the interfaces and IP addresses of the container(s) involved? Overlay networks add another whole level of complexity if that networking is different than the host networking.

There is no overlay network here -- this is not Kubernetes. We're running Podman-HPC (https://github.com/NERSC/podman-hpc) using Slurm. The container sees the host network.

[JBlaschke]

Your description of the problem lists 6 private IP addresses and 1 public IP address

Oh one more thing: of the interfaces listed in my example: OpenMPI definitely should not use nmn0 -- that network is for node management only (communicating over it will be slow, and can cause problems for other users and the sysadmins)

[jsquyres]

Since each node has one NIC/GPU and NUMA node, the ideal case would be for OpenMPI to use the "nearest" NIC with respect to the rank's NUMA domain.

I don't know how to parse this sentence; it seems to contradict itself. The first part of the sentence says that there's one NIC/GPU and NUMA node, but then the second part of the sentence implies that the Open MPI process should use the nearest NIC according to its NUMA domain.

Later in the text, you specifically mention that there are 4 NICs and 4 GPUs; I infer that this means that there are 4 NUMA domains, too.

I don't see why we would want 4 subnets.

I think you should check into how the Linux kernel handles IP traffic from multiple interfaces that are all on the same subnet. It's been a little while since I've poked around in that area, but it used to be true that all outgoing traffic to that subnet would go out the "first" interface on that subnet. I.e., all your outgoing traffic -- for that subnet -- would go out a single NIC. Perhaps the Linux kernel IP stack has gotten smarter about this over time, but I think this kind of use case is not(or at least: has not historically been) well represented in the Linux kernel IP space. Multiple interfaces on the same subnet were more intended to be used for bonding and the like, not NUMA-friendly transfers.

Also, this begs the larger question: why are you using TCP? Assuming you're using NVIDIA NICs and GPUs, shouldn't you be using UCX? UCX will handle all the interface selection and pinning, etc. It will also handle all the RDMA and GPUDirect stuff (which TCP won't).

[rhc54]

UCX will handle all the interface selection and pinning, etc.

Except for the OOB subsystem, of course.

[JBlaschke]

Later in the text, you specifically mention that there are 4 NICs and 4 GPUs; I infer that this means that there are 4 NUMA domains, too.

Oops, my response got mangled in edits. Apologies. Each node on Perlmutter has 1 CPU, 4 GPUs, and 4 NICs. The NICs and GPUs each are on a single NUMA node. (https://docs.nersc.gov/systems/perlmutter/architecture/) So yes, you have 4 NUMA domains, each with their own GPU and NIC.

Here's lstopo:

[JBlaschke]

I think you should check into how the Linux kernel handles IP traffic from multiple interfaces that are all on the same subnet. It's been a little while since I've poked around in that area, but it used to be true that all outgoing traffic to that subnet would go out the "first" interface on that subnet. I.e., all your outgoing traffic -- for that subnet -- would go out a single NIC. Perhaps the Linux kernel IP stack has gotten smarter about this over time, but I think this kind of use case is not(or at least: has not historically been) well represented in the Linux kernel IP space. Multiple interfaces on the same subnet were more intended to be used for bonding and the like, not NUMA-friendly transfers.

Clearly, you can bind to multiple interfaces, and send traffic via those ... I've done that in plenty of applications. So I suspect that the kernel's IP stack has gotten smarter.

Anyway, you might be right. But HPE won't change the network architecture of their flagship high-speed network over this. So I think speculating over how to arrange interfaces and subnets is moot -- seeing that we won't be able to change that.

[JBlaschke]

Assuming you're using NVIDIA NICs and GPUs, shouldn't you be using UCX? UCX will handle all the interface selection and pinning, etc. It will also handle all the RDMA and GPUDirect stuff (which TCP won't).

It's Nvidia GPUs and HPE NICs -- again here's the link: https://docs.nersc.gov/systems/perlmutter/architecture/. Also since HPE bought Cray, I don't know how well supported UCX is right now (with respect to Casini NICs. FTR: I like UCX, and would prefer to use it too).

HPE provides libfabric, built against their CXI (https://docs.open-mpi.org/en/main/tuning-apps/networking/ofi.html#what-are-the-libfabric-ofi-components-in-open-mpi) and their own MPI implementation.

That's all the official support from the vendor that I am aware of...

[JBlaschke]

Also, this begs the larger question: why are you using TCP?

Ah! Fair point. We are straying very far from the original point of this issue though. But for the sake of completeness, I will round out this discussion:

My goal is to help users be productive on our systems. Almost always that means minimizing the time it takes to have meaningful scientific data. Depending on the user's problem that can involve everything from performance tuning at scale, or just getting a pre-built executable to work at all.

So, if raw performance was my goal, right now I would be using the vendor's recommended transport library (which in this case would be Cray MPICH) -- even if I'm partial to OpenMPI 😉 Often though, some users do not have the wherewithal to compile a large application using a system-specific toolchain.

In an ideal world ABIs and standards would be mature enough that things like CXI could be resolved dynamically, and MPI "just works" -- in that case, the vendor could just provide something like and OFI provider, we would provide sensible configurations, and an application built against any MPI implementation would pick those up.

For now, this reality isn't here (yet). In its absence, TCP has established itself as the least common denominator that usually "just works". For many users, that is enough. (or at least the performance gains are not worth the effort of rebuilding their applications).

We are working with the CP2K developers, as well as Nvidia, to build a CP2K container that uses HPE CXI. I am happy to discuss this -- and other HPE-related topics -- further. Eventually though, I would like to return to the original point of this issue: that OpenMPI's TCP-based BTL is not detecting all interface addresses.

[rhc54]

Yeah, Slingshot NICs means you cannot use UCX - must use libfabric. However, I believe OMPI does support Slingshot operations - in fact, last I checked OMPI v5 is running on Perlmutter.

I'm curious, though - is mpirun finding the right interface for its OOB subsystem? Or is that also a problem?

[JBlaschke]

However, I believe OMPI does support Slingshot operations - in fact, last I checked OMPI v5 is running on Perlmutter.

Correct! But that requires OpenMPI to be built with libfabric support (and AFAIK, it needs to be the right version of libfabric -- I might be wrong though). These have to be configured at compile time, and are missing from the Nvidia container. We are working with Nvidia (who built https://catalog.ngc.nvidia.com/orgs/hpc/containers/cp2k ) to see if they can update their image (and upgrade to OMPI v5 while their at it).

I'm curious, though - is mpirun finding the right interface for its OOB subsystem? Or is that also a problem?

We're running the application with Slurm and PMI2: srun --mpi pmi2 .... My knowledge is limited in this area, I always assumed that this means OMPI using PMI2 to wire up the coms, and is not using its OOB subsystem?

[rhc54]

We're running the application with Slurm and PMI2: srun --mpi pmi2 .... My knowledge is limited in this area, I always assumed that this means OMPI using PMI2 to wire up the coms, and is not using its OOB subsystem?

Correct. However, you have to use PMIx with OMPI v5 (no pmi2 support any more).

see if they can update their image (and upgrade to OMPI v5 while their at it)

Those folks at NVIDIA are rather fixated on UCX, so that could be a problem convincing them. Truly wish you luck on it! 😄