opal_show_help() is not aggregating

[jsquyres]

In some IM discussions with @rhc54, we discovered that opal_show_help() message aggregation is no longer working (on both master and v2.x). This is a regression compared to v1.10.x.

There's two reasons it isn't working:

a) In orte/util/orte_show_help.c, the following checks are done to see if we should render the help message locally or relay it up to the HNP:

    if (ORTE_PROC_IS_HNP || ORTE_PROC_IS_TOOL ||
        orte_standalone_operation ||
        NULL == orte_rml.send_buffer_nb ||
        NULL == orte_routed.get_route ||
        NULL == orte_process_info.my_hnp_uri) {
        // do local render
    } else {
        // relay to HNP
    }

orte_rml.send_buffer_nb and orte_routed.get_route will now always be NULL in MPI procs.

b) Even if we delete these 2 checks (in two places, BTW), orte_show_help tries to use the RML to relay the help message up to the HNP. But there's no RML.

@rhc54 says it will take a bit of work to fix this.

[hppritcha]

Hmmm... okay is this another thing to just document in this release? It seems along the same lines as the debugger launch problem.

[jsquyres]

@rhc54 Do you have an estimate on a fix for this? This is a dumb, but kinda well-known feature in Open MPI... (i.e., users like it quite a bit)

[rhc54]

sigh...let me try to get it for you this weekend

[rhc54]

@jsquyres I've looked at this, and there is the quick fix - or the probably better fix.

Quick fix: just restore the usock oob component and have apps use it again.

Maybe better fix: we currently have all opal_output being directed out the stddiag fd that we hold on each process (we tie stddiag to stderr if we are direct launched). When we go thru show_help, we instead pack a message with the filename, text ident, and message and then send the buffer to the HNP.

What we could do is:

• redirect opal_output to stderr. We did this originally so that users could separate our diagnostic messages from their stderr stream, even allowing them to dump the two streams into different files. Moving opal_output to stderr would eliminate that feature, but allows us to separate show_help.
• have show help print to stddiag in a formatted manner. We already have xml format capability in there, so we could just use that to help tag things correctly. Either way, the daemon captures this stream separately, and so we could have it pack the "old" message format and send it to the show_help RML tag on the HNP.

So I guess this boils down to trading the usock component for the feature of separating opal_output from stderr. I honestly don't care either way - eliminating usock saved us an fd/process plus some memory footprint, but maybe it's more pain than it's worth.

Let me know which way you want to go. I personally am leaning towards just restoring usock, and will prep a PR for that purpose.