OMPI v3.0: pmix install dirs info is not propagated to orteds

[artpol84]

Open MPI version

gitclone ompi v3.0.x (2f13cce)

Details of the problem

If OMPI installation is moved to a different location OPAL_PREFIX env var is used to identify that and installdirs/env is handling this correctly.
PMIx also has MCA infrastructure and similar installdirs component. in PMIx PMIX_INSTALL_PREFIX playing the role of OPAL_PREFIX. The important difference is that OPAL_PREFIX is an OMPI variable and it gets propagated to orte daemons, i.e.:

[cn01:17570] [[10714,0],0] plm:rsh: final template argv:
/usr/bin/ssh <template> \
	OPAL_PREFIX=<ompi-path>/ompi-v3.0.x ; export OPAL_PREFIX; \
	PATH=<ompi-path>/ompi-v3.0.x/bin:$PATH ; export PATH ; \
	LD_LIBRARY_PATH=<ompi-path>/ompi-v3.0.x/lib:$LD_LIBRARY_PATH ; export LD_LIBRARY_PATH ; \
	DYLD_LIBRARY_PATH=<ompi-path>/ompi-v3.0.x/lib:$DYLD_LIBRARY_PATH ; export DYLD_LIBRARY_PATH ;   \
	<ompi-path>/ompi-v3.0.x/bin/orted -mca orte_debug_daemons "1" -mca ess "env" -mca ess_base_jobid "702152704" \
	-mca ess_base_vpid "<template>" -mca ess_base_num_procs "2" -mca orte_node_regex "cn01,cn02@0(2)" \
	-mca orte_hnp_uri "702152704.0;tcp://<IP1>,<IP2>;ud://<UD>" -mca coll_hcoll_enable "1" -mca pml "yalla" \
	--mca plm_base_verbose "100" -mca plm "rsh" -mca rmaps_base_mapping_policy "node" \
	-mca hwloc_base_binding_policy "core" -mca rmaps_base_display_map "1"

As it can be seen there PMIX_INSTALL_PREFIX is not propagated so orteds are failing with:

--------------------------------------------------------------------------
Sorry!  You were supposed to get help about:
    listener-thread-start
But I couldn't open the help file:
    <OLD-INSTALLATION_PATH>/ompi-v3.0.x/share/pmix/help-pmix-server.txt: No such file or directory.  Sorry!

Because they can't find PMIx mca ptl components.

[artpol84]

@rhc54 I know that on the dev meeting we had discussion about variable propagation. Do we have any news on that?

[artpol84]

One of the obvious solutions is to check if OPAL_PREFIX is set in pmix2x_server_init and if PMIX_INSTALL_PREFIX is not set - export necessary PMIX variables.
Something like this may needs to be done for the client as well.
However I wonder if there is more generic approach to this.

[rhc54]

We said at the meeting that we would create some kind of registration mechanism to cover these things, but that won't be ready for awhile and certainly wouldn't go into 3.0. For now, the only real solution is to manually do these things in the schizo/ompi component. I can add some code to cover it.

[artpol84]

Ok, thank you.

[rhc54]

Actually, one correction: we only forward OPAL_PREFIX for the plm/rsh component. So the addition will occur there.

[artpol84]

I only checked with pml/rsh but I it would be good to make sure that others are working fine as well.

[rhc54]

There actually isn't any way to forward something in the other methods - we have to rely solely on the configuration of the resource manager. Usually that config will forward nearly everything - i.e., the config generally doesn't scan for particular envar patterns like OPAL_. However, you are correct that users know to set the OPAL envar and expect it to cover everything. Probably the easiest solution is to just special case it and see if the daemon spots it, and then add the corresponding envar.

Ugly, but likely the only current solution.

[artpol84]

To clarify: in the issue description PMIX_INSTALL_PREFIX variable was manually set by me.