ORTE regex fails when nodenames include hyphens

[rhc54]

If a cluster is configured with - in the names of their nodes, then the ORTE regex parser gets confused and thinks the hyphen is part of a range designation. This leads to unusually large regex results and can cause odd behavior (e.g., unable to tree spawn).

Fortunately, we haven't seen this in 14 years, but Mellanox now has a cluster that hits the problem. No rules against such configuration, so probably something that should be addressed in case someone else does it.

[rhc54]

Just to further clarify: this problem will occur regardless of the plm launcher being used (e.g., when using mpirun under slurm) as we always try to pass the regex to the backend daemons on launch, and there is no guarantee that the length restriction will save us like it did in the referenced case.

[ggouaillardet]

@rhc54 i am a bit puzzled with this ...

i renamed my virtual cluster n-0, n-1 and so on, but was unable to reproduce the hang (this is not a large cluster, so the regex length is small enough.

there is a more obvious error when the regex is too long, and this does not involve - in the regex.

that can be simply reproduced with the inline patch below, and i can reproduce the issue when running

mpirun -npernode 1 -hostfile m -mca plm rsh -mca oob tcp -mca routed_radix 1 --mca plm_base_verbose 10 hostname

from n0 and my hostfile m contains

n1
n2

at first glance, i did not see n1 perform the phone home operation, but if you can point me to there, i will be happy to debug it.

diff --git a/orte/mca/plm/base/plm_base_launch_support.c b/orte/mca/plm/base/plm_base_launch_support.c
index 118813e..54930db 100644
--- a/orte/mca/plm/base/plm_base_launch_support.c
+++ b/orte/mca/plm/base/plm_base_launch_support.c
@@ -1573,6 +1573,7 @@ int orte_plm_base_orted_append_basic_args(int *argc, char ***argv,
         ORTE_ERROR_LOG(rc);
         return rc;
     }
+#if 0
     /* if this is too long, then we'll have to do it with
      * a phone home operation instead */
     if (strlen(param) < ORTE_MAX_REGEX_CMD_LENGTH) {
@@ -1582,6 +1583,7 @@ int orte_plm_base_orted_append_basic_args(int *argc, char ***argv,
         /* mark that the nidmap has been communicated */
         orte_nidmap_communicated = true;
     }
+#endif
     free(param);
 
     if (!orte_static_ports && !orte_fwd_mpirun_port) {

[rhc54]

If you only have two nodes in the hostfile, such as the Mellanox folks had, then the regex cannot be more than 1024 characters in length (which is the cmd_length limit). We run over 20k nodes with it and have never hit the limit.

I would look at the regex generator and see what it is doing wrong in that scenario.

[rhc54]

One place to start might be to simply print out the regex and see why it is so long...

[ggouaillardet]

ok, let me put it this way then,

i created a cluster whose nodes are called frontend, node-001 and node-002 and i am unable to reproduce the issue @karasevb is facing.

the regex (that can be seen with --mca plm_base_verbose 10) is a simple expansion of the host list, in this case it is frontend,node-001,node-002@0(3).

should we fix the real issue (e.g. the phone home does not work) before optimizing the regex generation ?

[rhc54]

I honestly don't think that is the real issue here. The regex isn't long enough to be getting rejected, so the question is: why is it failing for Mellanox when it doesn't fail for you?

Once we understand and fix that, then sure - we can look into why the alternative code path isn't working. I honestly don't remember the code path for that case, so it would take a little looking to see how it is supposed to work.

[ggouaillardet]

@rhc54 i am fine with that.

@karasevb is Open MPI installed on NFS ? if not, can you please double check the very same Open MPI is deployed on all the nodes ? i have a real hard time trying to reproduce the issue, and all i can think of is (as @rhc54 previously suggested it) some node is running an older version of v3.0.x without the fix.

[rhc54]

Another thing we could do is provide a patch for orte/util/regex.c that adds some debug and have @karasevb run it to see what is going on over there.

[rhc54]

@ggouaillardet Actually, I just realized that your output indeed shows the problem. The regex should have compressed node-001 and node-002 into a regex like node-[3:1-2]. The hyphen is confusing it into thinking every node name is unique and cannot be compressed.

This is why Mellanox is failing. They have a lot of nodes (we don't filter them when making the original regex), and the regex generator thinks they are all unique names. Otherwise, it would have output something like "node-[3:1-999]".

[ggouaillardet]

@rhc54 minor detail, the regex should be node-00[3:1-2]
about the "lot of nodes" thing, are you saying that even with a two line hostfile, and forcing the plm/rsh, the regex includes all the (100 iirc) nodes from the slurm allocation ?