Inconsistent segfaults on non-uniform topology

[Chrismarsh]

I'm seeing some very strange transient segfaults, where I can run this code 10 times and it errors inconsistently. I am building openmpi and prrte via spack from source.

 mpirun -n 1000 true
[myhost-157:1301000] *** Process received signal ***
[myhost-157:1301000] Signal: Segmentation fault (11)
[myhost-157:1301000] Signal code: Address not mapped (1)
[myhost-157:1301000] Failing at address: 0x1b8
[myhost-157:1301000] [ 0] /lib64/libc.so.6(+0x3e6f0)[0x7efc71bd66f0]
[myhost-157:1301000] [ 1] /spack/opt/linux-icelake/hwloc-2.13.0-enoxcy27v34v6bjqlcqpxvq22h3aw2ga/lib/libhwloc.so.15(hwloc_topology_get_allowed_cpuset+0x0)[0x7efc71db0ee0]
[myhost-157:1301000] [ 2] /spack/opt/linux-icelake/prrte-4.1.0-ip2ammxivjakk24szc2c6kka2cjc4gte/lib/libprrte.so.3(prte_hwloc_base_filter_cpus+0x8c)[0x7efc729b615c]
[myhost-157:1301000] [ 3] /spack/opt/linux-icelake/prrte-4.1.0-ip2ammxivjakk24szc2c6kka2cjc4gte/lib/libprrte.so.3(prte_plm_base_daemon_callback+0x1b08)[0x7efc72a16da8]
[myhost-157:1301000] [ 4] /spack/opt/linux-icelake/prrte-4.1.0-ip2ammxivjakk24szc2c6kka2cjc4gte/lib/libprrte.so.3(prte_rml_base_process_msg+0x249)[0x7efc729bb7a9]
[myhost-157:1301000] [ 5] /spack/opt/linux-icelake/libevent-2.1.12-cchgrwhqu4eqr4v5stf2x2wklzwv3sn3/lib/libevent_core-2.1.so.7(+0x1dff2)[0x7efc720c4ff2]
[myhost-157:1301000] [ 6] /spack/opt/linux-icelake/libevent-2.1.12-cchgrwhqu4eqr4v5stf2x2wklzwv3sn3/lib/libevent_core-2.1.so.7(event_base_loop+0x47f)[0x7efc720c56bf]
[myhost-157:1301000] [ 7] /spack/opt/linux-icelake/prrte-4.1.0-ip2ammxivjakk24szc2c6kka2cjc4gte/lib/libprrte.so.3(prte+0x23d4)[0x7efc72992634]
[myhost-157:1301000] [ 8] /lib64/libc.so.6(+0x29590)[0x7efc71bc1590]
[myhost-157:1301000] [ 9] /lib64/libc.so.6(__libc_start_main+0x80)[0x7efc71bc1640]
[myhost-157:1301000] [10] prterun[0x401075]
[myhost-157:1301000] *** End of error message ***
Segmentation fault (core dumped)

running with mpirun --debug-daemons --prtemca plm_base_verbose 100 -n 1000 true
I have quite a bit of heterogeneity

NUMA[35,43,22,3:0]
NUMA[40,43,17,3:0]
NUMA[2:43,14,3:0]

and

RECEIVED TOPOLOGY SIG ... FROM NODE myhostcn-073
NEW TOPOLOGY - ADDING SIGNATURE

Then another node:

RECEIVED TOPOLOGY SIG ... FROM NODE myhostcn-128
TOPOLOGY SIGNATURE ALREADY RECORDED

and then the segfault. I'm wondering if you've seen this issue? The issue open-mpi/ompi#13357 seems similar but prrte 4.1 doesn't seem to resolve it fully for me.

[rhc54]

Try adding --hetero-nodes to your cmd line and see if that resolves it. We have no reports of the problem, but that isn't terribly surprising as hetero nodes is a somewhat unusual situation.

[rhc54]

Just glancing at the code, I can see how the segfault might happen. Setting that cmd line option should provide a workaround, but I'll also give you a patch to try.

[rhc54]

Try this:

diff --git a/src/mca/plm/base/plm_base_launch_support.c b/src/mca/plm/base/plm_base_launch_support.c
index c383abacb9..9d696c0e66 100644
--- a/src/mca/plm/base/plm_base_launch_support.c
+++ b/src/mca/plm/base/plm_base_launch_support.c
@@ -1747,12 +1747,20 @@ void prte_plm_base_daemon_callback(int status, pmix_proc_t *sender, pmix_data_bu
                 if (NULL == t->topo && NULL != topo) {
                     t->topo = topo;
                 }
-                /* update the node's available processors */
-                if (NULL != daemon->node->available) {
-                    hwloc_bitmap_free(daemon->node->available);
+                /* the topology might still be NULL because a daemon with
+                 * this signature arrived first, but didn't provide a
+                 * topology and so we are waiting for that request to
+                 * return. So we have to protect ourselves here. The
+                 * available bitmap will be set once the full topology
+                 * is received */
+                if (NULL != t->topo) {
+                    /* update the node's available processors */
+                    if (NULL != daemon->node->available) {
+                        hwloc_bitmap_free(daemon->node->available);
+                    }
+                    daemon->node->available = prte_hwloc_base_filter_cpus(t->topo);
+                    prte_hwloc_base_setup_summary(t->topo);
                 }
-                daemon->node->available = prte_hwloc_base_filter_cpus(t->topo);
-                prte_hwloc_base_setup_summary(t->topo);
                 break;
             }
         }

[Chrismarsh]

Thanks for the fast turn around!
With mpirun --hetero-nodes I can't reproduce crashes, it seems to solve the problem.

With the patch, it no longer segfaults, but I 100% of the time now get this

------------------------------------------------------------
A daemon failed to report back after being spawned. This could
be due to several factors, including inability to find the
daemon executable, or the executable was unable to find its
required supporting libraries. In some cases, the daemon was
able to execute, but was unable to complete a TCP connection
back to prterun:

  Local host:    myhostcn-076
  Remote host:  myhostcn-058-ib
  Daemon exit status: 255

This may also be caused by a firewall on the remote host. Please
check that any firewall (e.g., iptables) has been disabled and
try again.
------------------------------------------------------------

[rhc54]

Can you please run the patch with the plm verbosity again?

[Chrismarsh]

ah, my apologies @rhc54: I forgot to build OMPI with PBS support. I was too excited to try the patch!

It still segfaults, although with much less regularity. It is also in a different place now:

[myhostn-034:1137657] Signal: Segmentation fault (11)
[myhostn-034:1137657] Signal code: Address not mapped (1)
[myhostn-034:1137657] Failing at address: (nil)
[myhostn-034:1137657] [ 0] /lib64/libc.so.6(+0x3e6f0)[0x7f82e14706f0]
[myhostn-034:1137657] [ 1] /project/spack/opt/linux-icelake/hwloc-2.13.0-enoxcy27v34v6bjqlcqpxvq22h3aw2ga/lib/libhwloc.so.15(hwloc_bitmap_copy+0x11)[0x7f82e1659b51]
[myhostn-034:1137657] [ 2] /project/spack/opt/linux-icelake/prrte-4.1.0-44q4wjwgkd3hhlalp74crjrlyg52k47p/lib/libprrte.so.3(prte_rmaps_base_get_target_nodes+0xcf0)[0x7f82e238ecc0]
[myhostn-034:1137657] [ 3] /project/spack/opt/linux-icelake/prrte-4.1.0-44q4wjwgkd3hhlalp74crjrlyg52k47p/lib/libprrte.so.3(+0x11af21)[0x7f82e2393f21]
[myhostn-034:1137657] [ 4] /project/spack/opt/linux-icelake/prrte-4.1.0-44q4wjwgkd3hhlalp74crjrlyg52k47p/lib/libprrte.so.3(prte_rmaps_base_map_job+0x263e)[0x7f82e238b69e]
[myhostn-034:1137657] [ 5] /project/spack/opt/linux-icelake/libevent-2.1.12-cchgrwhqu4eqr4v5stf2x2wklzwv3sn3/lib/libevent_core-2.1.so.7(+0x1dff2)[0x7f82e195cff2]
[myhostn-034:1137657] [ 6] /project/spack/opt/linux-icelake/libevent-2.1.12-cchgrwhqu4eqr4v5stf2x2wklzwv3sn3/lib/libevent_core-2.1.so.7(event_base_loop+0x47f)[0x7f82e195d6bf]
[myhostn-034:1137657] [ 7] /project/spack/opt/linux-icelake/prrte-4.1.0-44q4wjwgkd3hhlalp74crjrlyg52k47p/lib/libprrte.so.3(prte+0x3f6f)[0x7f82e22ea30f]
[myhostn-034:1137657] [ 8] /lib64/libc.so.6(+0x29590)[0x7f82e145b590]
[myhostn-034:1137657] [ 9] /lib64/libc.so.6(__libc_start_main+0x80)[0x7f82e145b640]
[myhostn-034:1137657] [10] prterun[0x401075]
[myhostn-034:1137657] *** End of error message ***

full logs from mpirun --debug-daemons --prtemca plm_base_verbose 100 -n 1000 true 2>&1 | tee prrte.txt
prrte.txt

Maybe helpful/interesting is that OMPI 4.x launched on the same machines has the following (ancient version) hwloc error about the bitmaps:

  Error message:     hwloc_set_cpubind returned "Error" for bitmap "0,256"
  Location:          rtc_hwloc.c:382

[rhc54]

Thanks - that helped track it down. Please replace the prior patch with this one:

diff --git a/src/mca/plm/base/plm_base_launch_support.c b/src/mca/plm/base/plm_base_launch_support.c
index c383abacb9..202bccd933 100644
--- a/src/mca/plm/base/plm_base_launch_support.c
+++ b/src/mca/plm/base/plm_base_launch_support.c
@@ -1690,14 +1690,7 @@ void prte_plm_base_daemon_callback(int status, pmix_proc_t *sender, pmix_data_bu
             sig = NULL;  // protect from release
             dc->daemon = daemon;
             pmix_list_append(&prte_plm_globals.daemon_cache, &dc->super);
-            free(nodename);
-            nodename = NULL;
-            idx = 1;
-            ret = PMIx_Data_unpack(NULL, buffer, &dname, &idx, PMIX_PROC);
-            if (PMIX_SUCCESS != ret) {
-                break;
-            }
-            continue;
+            goto CLEANUP;
         }

         if (prte_hetero_nodes) {
@@ -1710,21 +1703,10 @@ void prte_plm_base_daemon_callback(int status, pmix_proc_t *sender, pmix_data_bu
             daemon->node->topology = t;
             daemon->node->available = prte_hwloc_base_filter_cpus(t->topo);
             prte_hwloc_base_setup_summary(t->topo);
-            // cannot have cached daemons
-            free(nodename);
-            nodename = NULL;
-            idx = 1;
-            ret = PMIx_Data_unpack(NULL, buffer, &dname, &idx, PMIX_PROC);
-            if (PMIX_SUCCESS != ret &&
-                PMIX_ERR_UNPACK_READ_PAST_END_OF_BUFFER != ret) {
-                PMIX_ERROR_LOG(ret);
-                PRTE_ACTIVATE_JOB_STATE(jdatorted, PRTE_JOB_STATE_FAILED_TO_START);
-                return;
-            }
             jdatorted->num_reported++;
             jdatorted->num_daemons_reported++;
             progress_daemons(jdatorted, show_progress);
-            continue;
+            goto CLEANUP;
         }

         /* do we already have this topology from some other node? */
@@ -1747,13 +1729,23 @@ void prte_plm_base_daemon_callback(int status, pmix_proc_t *sender, pmix_data_bu
                 if (NULL == t->topo && NULL != topo) {
                     t->topo = topo;
                 }
-                /* update the node's available processors */
-                if (NULL != daemon->node->available) {
-                    hwloc_bitmap_free(daemon->node->available);
+                /* the topology might still be NULL because a daemon with
+                 * this signature arrived first, but didn't provide a
+                 * topology and so we are waiting for that request to
+                 * return. So we have to protect ourselves here. The
+                 * available bitmap will be set once the full topology
+                 * is received */
+                if (NULL != t->topo) {
+                    /* update the node's available processors */
+                    if (NULL != daemon->node->available) {
+                        hwloc_bitmap_free(daemon->node->available);
+                    }
+                    daemon->node->available = prte_hwloc_base_filter_cpus(t->topo);
+                    prte_hwloc_base_setup_summary(t->topo);
+                    break;
+                } else {
+                    goto CLEANUP;
                 }
-                daemon->node->available = prte_hwloc_base_filter_cpus(t->topo);
-                prte_hwloc_base_setup_summary(t->topo);
-                break;
             }
         }

@@ -1837,17 +1829,8 @@ void prte_plm_base_daemon_callback(int status, pmix_proc_t *sender, pmix_data_bu
                 }
                 /* we will count this node as completed
                  * when we get the full topology back */
-                if (NULL != nodename) {
-                    free(nodename);
-                    nodename = NULL;
-                }
-            }
-            idx = 1;
-            ret = PMIx_Data_unpack(NULL, buffer, &dname, &idx, PMIX_PROC);
-            if (PMIX_SUCCESS != ret) {
-                break;
             }
-            continue;
+            goto CLEANUP;

         } else {  // not daemon rank=1
             if (!found) {
@@ -1856,7 +1839,8 @@ void prte_plm_base_daemon_callback(int status, pmix_proc_t *sender, pmix_data_bu
                                     "%s NEW TOPOLOGY - ADDING SIGNATURE",
                                     PRTE_NAME_PRINT(PRTE_PROC_MY_NAME));
                 t = PMIX_NEW(prte_topology_t);
-                t->sig = strdup(sig);
+                t->sig = sig;
+                sig = NULL;
                 t->index = pmix_pointer_array_add(prte_node_topologies, t);
                 daemon->node->topology = t;
                 if (NULL != topo) {
@@ -1874,7 +1858,8 @@ void prte_plm_base_daemon_callback(int status, pmix_proc_t *sender, pmix_data_bu
                     // add this daemon to our cache
                     dc = PMIX_NEW(dcaddy_t);
                     dc->rank = daemon->name.rank;
-                    dc->sig = strdup(sig);
+                    dc->sig = sig;
+                    sig = NULL;
                     dc->t = t;
                     dc->daemon = daemon;
                     pmix_list_append(&prte_plm_globals.daemon_cache, &dc->super);
@@ -1899,20 +1884,7 @@ void prte_plm_base_daemon_callback(int status, pmix_proc_t *sender, pmix_data_bu
                     /* we will count this node as completed
                      * when we get the full topology back */
                 }
-                if (NULL != nodename) {
-                    free(nodename);
-                    nodename = NULL;
-                }
-                if (NULL != sig) {
-                    free(sig);
-                    sig = NULL;
-                }
-                idx = 1;
-                ret = PMIx_Data_unpack(NULL, buffer, &dname, &idx, PMIX_PROC);
-                if (PMIX_SUCCESS != ret) {
-                    break;
-                }
-                continue;
+                goto CLEANUP;
             }
             // Signature already found, we can mark this node as completed now
             jdatorted->num_reported++;

OMPI 4.x launched on the same machines has the following (ancient version) hwloc error

Yeah, that version won't handle this level of heterogeneity.

[Chrismarsh]

Yeah, that version won't handle this level of heterogeneity.

Good to know thanks

I just triple checked that I did apply this patch correctly (spack confirms a patch with the correct checksum was applied). Alas, the segfault is still there. And it's once again into a different location:

[myhost-031:329606] [prterun-myhost-031-329606@0,0] RECEIVED TOPOLOGY SIG PKG[100,0];NUMA[42,43,15,3:0];L3[100,0];L2[100:1];L1[100:1]:x86_64:le FROM NODE myhost-031
[myhost-031:329606] [prterun-myhost-031-329606@0,0] TOPOLOGY SIGNATURE ALREADY RECORDED IN POSN 0
[myhost-031:329606] [prterun-myhost-031-329606@0,0] plm:base:prted_daemon_cback processing cached daemon 7
[myhost-031:329606] [prterun-myhost-031-329606@0,0] plm:base:prted_daemon_cback processing cached daemon 2
[myhost-031:329606] [prterun-myhost-031-329606@0,0] plm:base:prted_daemon_cback processing cached daemon 3
[myhost-031:329606] *** Process received signal ***
[myhost-031:329606] Signal: Segmentation fault (11)
[myhost-031:329606] Signal code: Address not mapped (1)
[myhost-031:329606] Failing at address: 0x1b8
[myhost-031:329606] [ 0] /lib64/libc.so.6(+0x3e6f0)[0x7fe36d62a6f0]
[myhost-031:329606] [ 1] /project/spack/opt/linux-icelake/hwloc-2.13.0-z7uayzzlt3ghph2ucjfjcw54uzlvt7du/lib/libhwloc.so.15(hwloc_topology_get_allowed_cpuset+0x0)[0x7fe36d808020]
[myhost-031:329606] [ 2] /project/spack/opt/linux-icelake/prrte-4.1.0-2uq3wjnz4lp6mt6qfjzpetstsca6bz7u/lib/libprrte.so.3(prte_hwloc_base_filter_cpus+0x8c)[0x7fe36dcb00dc]
[myhost-031:329606] [ 3] /project/spack/opt/linux-icelake/prrte-4.1.0-2uq3wjnz4lp6mt6qfjzpetstsca6bz7u/lib/libprrte.so.3(prte_plm_base_daemon_callback+0x1bf7)[0x7fe36dd10b17]
[myhost-031:329606] [ 4] /project/spack/opt/linux-icelake/prrte-4.1.0-2uq3wjnz4lp6mt6qfjzpetstsca6bz7u/lib/libprrte.so.3(prte_rml_base_process_msg+0x249)[0x7fe36dcb5729]
[myhost-031:329606] [ 5] /project/spack/opt/linux-icelake/libevent-2.1.12-cchgrwhqu4eqr4v5stf2x2wklzwv3sn3/lib/libevent_core-2.1.so.7(+0x1dff2)[0x7fe36db1aff2]
[myhost-031:329606] [ 6] /project/spack/opt/linux-icelake/libevent-2.1.12-cchgrwhqu4eqr4v5stf2x2wklzwv3sn3/lib/libevent_core-2.1.so.7(event_base_loop+0x47f)[0x7fe36db1b6bf]
[myhost-031:329606] [ 7] /project/spack/opt/linux-icelake/prrte-4.1.0-2uq3wjnz4lp6mt6qfjzpetstsca6bz7u/lib/libprrte.so.3(prte+0x23d4)[0x7fe36dc8c5b4]
[myhost-031:329606] [ 8] /lib64/libc.so.6(+0x29590)[0x7fe36d615590]
[myhost-031:329606] [ 9] /lib64/libc.so.6(__libc_start_main+0x80)[0x7fe36d615640]
[myhost-031:329606] [10] /project/spack/opt/linux-icelake/prrte-4.1.0-2uq3wjnz4lp6mt6qfjzpetstsca6bz7u/bin/prterun[0x401075]
[myhost-031:329606] *** End of error message ***

prrte_2.txt

I don't know if it matters, but this version is building without the tm integration due to the library issue noted in #2425 (comment), so launching with /FQP/to/mpirun --map-by node --prtemca plm_ssh_args "-o SendEnv=PBS_JOBID" "$@"

Is there any other information I can provide you to help debug?
n.b., I will be afk for a week

[rhc54]

The --without-tm doesn't matter here. We seem to be having a problem with the topology caching code in the daemon callback when encountering a lot of heterogeneity.

I'll play with this for a bit while you are gone and see if I can conjure a reproducer - let's pick it up again when you return.

Have a good time afk!