Open MPI on bonded interface issues

[SharonBrunett]

I would like to run OpenMPI 4.1.1 on a collection of nodes using a bonded interface.

Small node counts (e.g. 6) sometimes works but larger node counts (e.g. 25) most always fail. A race condition or non deterministic behavior of some kind. Here's the error I get

Open MPI detected an inbound MPI TCP connection request from a peer
that appears to be part of this MPI job (i.e., it identified itself as
part of this Open MPI job), but it is from an IP address that is
unexpected.  This is highly unusual.

The inbound connection has been dropped, and the peer should simply
try again with a different IP interface (i.e., the job should
hopefully be able to continue).

  Local host:          node1390
  Local PID:           389375
  Peer hostname:       node836 ([[19416,1],0])
  Source IP of socket: 10.9.4.86
  Known IPs of peer:   
	10.14.4.86

Here's node836's, node1390's and node1394's "ip addr" Note the bond on 1390 and 1394, I launch the job from node836.

(openmpi-test) [sharon.brunett@**node836 openmpi_cit_test]$ ip addr**
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000
    link/ether 98:40:bb:bc:90:85 brd ff:ff:ff:ff:ff:ff
    inet 10.9.4.86/16 brd 10.9.255.255 scope global noprefixroute em1
       valid_lft forever preferred_lft forever
    inet 10.14.4.86/16 brd 10.14.255.255 scope global noprefixroute em1
       valid_lft forever preferred_lft forever
3: em2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 98:40:bb:bc:90:88 brd ff:ff:ff:ff:ff:ff

**[sharon.brunett@node1390 ~]$ ip addr**
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 0c:c4:7a:97:05:f4 brd ff:ff:ff:ff:ff:ff
3: eno2: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 9000 qdisc mq master bond0 state DOWN group default qlen 1000
    link/ether 0c:c4:7a:97:05:f5 brd ff:ff:ff:ff:ff:ff
4: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
    link/ether 0c:c4:7a:97:05:f4 brd ff:ff:ff:ff:ff:ff
    inet 10.9.6.140/16 brd 10.9.255.255 scope global bond0
       valid_lft forever preferred_lft forever
    inet 10.14.6.140/16 brd 10.14.255.255 scope global bond0
       valid_lft forever preferred_lft forever

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 0c:c4:7a:97:03:c0 brd ff:ff:ff:ff:ff:ff
3: eno2: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 9000 qdisc mq master bond0 state DOWN group default qlen 1000
    link/ether 0c:c4:7a:97:03:c1 brd ff:ff:ff:ff:ff:ff
4: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
    link/ether 0c:c4:7a:97:03:c0 brd ff:ff:ff:ff:ff:ff
    inet 10.9.6.144/16 brd 10.9.255.255 scope global bond0
       valid_lft forever preferred_lft forever
    inet 10.14.6.144/16 brd 10.14.255.255 scope global bond0
       valid_lft forever preferred_lft forever

Here's the little test...if I drop the --barrier, the test does not generate errors (but I need the barrier for the larger code). I wonder why?

import sys
import logging as log
import time
import platform
from mpi4py import MPI as mpi

mpi_comm = mpi.COMM_WORLD
mpi_rank = mpi_comm.Get_rank()

log_format = "%(asctime)s [host {0} rank {1:d}] %(message)s".format(platform.node(), mpi_rank)
log.basicConfig(format=log_format, level=log.INFO)

def master():
    # rank=0 will just sit waiting for data and printing it as it comes

    while True:
        stuff = mpi_comm.gather(None, root=0)
        log.info('Got: %s', stuff)

def compute():
    # rank>0 will keep sending data periodically
    # at a rate proportional to (rank - 1)

    iteration = 0

    while True:
        stuff = 'rank {:d} at iteration {:d}'.format(mpi_rank, iteration)

        log.info('Sending: %s', stuff)
        t0 = time.time()
        mpi_comm.gather(stuff, root=0)
        t1 = time.time()
        log.info('gather() blocked for %f s', t1 - t0)

        sleep_time = 1 * mpi_rank
        log.info('Sleeping for %d s', sleep_time)
        time.sleep(sleep_time)

        iteration += 1


log.info('Ready')

if len(sys.argv) > 1 and sys.argv[1] == '--barrier':
    mpi_comm.Barrier()

if mpi_rank == 0:
    master()
else:
    compute()

Thanks for your inputs on what to try next.
Sharon

[jsquyres]

I must be missing something; I don't quite understand the rationale for your setup.

• 836: has 2 IPs on a single, non-bonded interface.
• 1390: has 2 IPs on a bonded interface
• 1394: has 2 IPs on a bonded interface

Why do your bonded interfaces have 2 IP addresses?

[SharonBrunett]

Jeff, We use the 10.9 interface for internal cluster traffic (e.g. HDFS) and the 10.14 interface to connect to login and head nodes. Do you think OpenMPI command line args could be tweaked to accommodate this configuration? Thanks, Sharon

…

________________________________ From: Jeff Squyres ***@***.***> Sent: Tuesday, May 18, 2021 7:20 AM To: open-mpi/ompi ***@***.***> Cc: SharonBrunett ***@***.***>; Author ***@***.***> Subject: Re: [open-mpi/ompi] Open MPI on bonded interface issues (#8967) I must be missing something; I don't quite understand the rationale for your setup. * 836: has 2 IPs on a single, non-bonded interface. * 1390: has 2 IPs on a bonded interface * 1394: has 2 IPs on a bonded interface Why do your bonded interfaces have 2 IP addresses? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#8967 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AEUTAJ3TWOT6YQ7S5VPA5WLTOJZR7ANCNFSM45BO7CFQ>.

[jsquyres]

Do you have different QoS configured in your fabric for the two different subnets?

I think in most cases that we have seen previously, there has been a different interface for each IP -- I don't think we've seen many cases where multiple IPs on a single interface have been used. Hence, this case is likely not well tested (and apparently doesn't work). It's not just a case of tweaking command line params -- it's a matter of changing how Open MPI detects and matches IP interfaces. This functionality appears in several different places in the Open MPI code base.

What happens if you --mca btl_tcp_if_exclude one of your networks (or include -- whichever way you want to go)? Note that you have to match the subnet correctly. Your subnet is 10.9.6.144/16, so you'd have to --mca btl_tcp_if_exclude 10.9.6.144/16 (not /24). I'm honestly not sure if this will work, but it's worth a try.

[SharonBrunett]

Hi Jeff,

Do you have different QoS configured in your fabric for the two different subnets?

No, the same physical interface is shared with no QoS setup. Should there be? I tried adding the appropriate subnet to exclude, for the four nodes (836,1390,1391, 1392) participating. Sadly, this doesn’t work. Below are the ip addr dumps from node[836,1390,1391,1392) and mpi runtime errors node836 ~]$ ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000 link/ether 98:40:bb:bc:90:85 brd ff:ff:ff:ff:ff:ff inet 10.9.4.86/16 brd 10.9.255.255 scope global noprefixroute em1 valid_lft forever preferred_lft forever inet 10.14.4.86/16 brd 10.14.255.255 scope global noprefixroute em1 valid_lft forever preferred_lft forever 3: em2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000 link/ether 98:40:bb:bc:90:88 brd ff:ff:ff:ff:ff:ff node1390 ~]$ ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: eno1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP group default qlen 1000 link/ether 0c:c4:7a:97:05:f4 brd ff:ff:ff:ff:ff:ff 3: eno2: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 9000 qdisc mq master bond0 state DOWN group default qlen 1000 link/ether 0c:c4:7a:97:05:f5 brd ff:ff:ff:ff:ff:ff 4: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000 link/ether 0c:c4:7a:97:05:f4 brd ff:ff:ff:ff:ff:ff inet 10.9.6.140/16 brd 10.9.255.255 scope global bond0 valid_lft forever preferred_lft forever inet 10.14.6.140/16 brd 10.14.255.255 scope global bond0 valid_lft forever preferred_lft forever node1391 ~]$ ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: eno1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP group default qlen 1000 link/ether 0c:c4:7a:97:02:f2 brd ff:ff:ff:ff:ff:ff 3: eno2: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 9000 qdisc mq master bond0 state DOWN group default qlen 1000 link/ether 0c:c4:7a:97:02:f3 brd ff:ff:ff:ff:ff:ff 4: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000 link/ether 0c:c4:7a:97:02:f2 brd ff:ff:ff:ff:ff:ff inet 10.9.6.141/16 brd 10.9.255.255 scope global bond0 valid_lft forever preferred_lft forever inet 10.14.6.141/16 brd 10.14.255.255 scope global bond0 valid_lft forever preferred_lft forever node1392 ~]$ ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: eno1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP group default qlen 1000 link/ether 0c:c4:7a:97:02:f4 brd ff:ff:ff:ff:ff:ff 3: eno2: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 9000 qdisc mq master bond0 state DOWN group default qlen 1000 link/ether 0c:c4:7a:97:02:f5 brd ff:ff:ff:ff:ff:ff 4: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000 link/ether 0c:c4:7a:97:02:f4 brd ff:ff:ff:ff:ff:ff inet 10.9.6.142/16 brd 10.9.255.255 scope global bond0 valid_lft forever preferred_lft forever inet 10.14.6.142/16 brd 10.14.255.255 scope global bond0 valid_lft forever preferred_lft forever ***@***.*** ~]$ mpirun --launch-agent /tmp/openmpi-test/bin/ortedostfile mpi_hosts_cit.txt --np 4 --map-by node --mca btl_tcp_if_exclude 10.9.6.142/16,10.9.6.141/16,10.9.6.140/16,10.9.4.86/16 --x PATH python -m mpi4py mpi_test.py --barrier 2021-05-20 12:13:48,284 [host node1390 rank 1] Ready 2021-05-20 12:13:48,285 [host node836 rank 0] Ready -------------------------------------------------------------------------- WARNING: Open MPI accepted a TCP connection from what appears to be a another Open MPI process but cannot find a corresponding process entry for that peer. This attempted connection will be ignored; your MPI job may or may not continue properly. Local host: node1390 PID: 923677 -------------------------------------------------------------------------- 2021-05-20 12:13:48,285 [host node1391 rank 2] Ready 2021-05-20 12:13:48,285 [host node1392 rank 3] Ready [node836:674881] 2 more processes have sent help message help-mpi-btl-tcp.txt / server accept cannot find guid [node836:674881] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages Thanks much for your inputs, Sharon

…

On May 20, 2021, at 6:07 AM, Jeff Squyres ***@***.*** ***@***.***>> wrote: different QoS configured in your fabric for the two different subnets

[jsquyres]

Do you have different QoS configured in your fabric for the two different subnets?

No, the same physical interface is shared with no QoS setup. Should there be?

• If there's no QoS differences between the subnets, and
• All the subnets are available on all the nodes, and
• On each node, the subnets share a single physical NIC port (corresponding to a single OS IP interface)

Then I'm not sure I see any benefit to having multiple IP's/subnets per interface. Is there something I'm missing?

If I'm not, then your simplest route would be to remove the 2nd IP interface from your nodes and just use a single subnet / IP address per interface. This Open MPI issue would then go away, too.

[SharonBrunett]

Jeff, So sorry for the delay. The benefit, in the case of our local configuration, for having multiple IPs/subnets per interface is so one IP interface can be used for internal cluster traffic (e.g. Ceph) to other cluster nodes, whereas the second IP interface is used for connections to the login/head nodes. Sharing a single physical NIC port is a function of the server’s hardware limitations. I experimented with adding QoS, no joy. A feature enhancement to OpenMPI in support of multiple IP's/subnets per interface would be much appreciated. Thanks for your advice on how to proceed. Sharon Brunett Caltech LIGO 626 395 6734 On May 21, 2021, at 10:57 AM, Jeff Squyres ***@***.******@***.***>> wrote: Do you have different QoS configured in your fabric for the two different subnets? No, the same physical interface is shared with no QoS setup. Should there be? * If there's no QoS differences between the subnets, and * All the subnets are available on all the nodes * On each node, the subnets share a single physical NIC port (corresponding to a single OS IP interface) Then I'm not sure I see any benefit to having multiple IP's/subnets per interface. Is there something I'm missing? If I'm not, then your simplest route would be to remove the 2nd IP interface from your nodes and just use a single subnet / IP address per interface. This Open MPI issue would then go away, too. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#8967 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AEUTAJ73TPQS6IUFSRN23IDTO2NJBANCNFSM45BO7CFQ>.