--prefix is not properly propagated to 3rd party libraries

[bosilca]

Thank you for taking the time to submit an issue!

Background information

What version of Open MPI are you using? (e.g., v3.0.5, v4.0.2, git branch name and hash, etc.)

OMPI main (fa08eac)

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

configure with --prefix. Changing the --prefix without wiping completely the build directory leads to inconsistencies as the prefix is not correctly propagated to some of the 3rd parties software (at least not to libevent). The opal_wrapper tool will contain an incorrect dependency to the old location of the 3rd party libraries.

In my case I changed the location from opt/master to opt/main (to reflect the branch renaming) and I get the following:

dyld[47461]: Library not loaded: /opt/ompi/master/debug/lib/libevent_core-2.1.7.dylib
  Referenced from: /opt/ompi/main/debug/bin/opal_wrapper
  Reason: tried: '/opt/ompi/master/debug/lib/libevent_core-2.1.7.dylib' (no such file), '/usr/local/lib/libevent_core-2.1.7.dylib' (no such file), '/usr/lib/libevent_core-2.1.7.dylib' (no such file)

Looking at the dependencies of opal_wrapper I see a mix of new and old path:

/opt/ompi/main/debug/bin/opal_wrapper:
	/opt/ompi/main/debug/lib/libopen-pal.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/opt/ompi/main/debug/lib/libpmix.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/opt/ompi/master/debug/lib/libevent_core-2.1.7.dylib (compatibility version 8.0.0, current version 8.1.0)
	/opt/ompi/master/debug/lib/libevent_pthreads-2.1.7.dylib (compatibility version 8.0.0, current version 8.1.0)
	/opt/ompi/main/debug/lib/libhwloc.15.dylib (compatibility version 21.0.0, current version 21.3.0)
	/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
	/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1858.112.0)
	@rpath/libgcc_s.1.1.dylib (compatibility version 1.0.0, current version 1.1.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.100.3)

If you are building/installing from a git clone, please copy-n-paste the output from git submodule status.

❯ git submodule status
 26ff1684378e1c5859e377249a3e78f287b04216 3rd-party/openpmix (v1.1.3-3523-g26ff1684)
 47b5ad1653123e984a644ef84b9a7aaf4e9b5fb3 3rd-party/prrte (psrvr-v2.0.0rc1-4355-g47b5ad1653)

Please describe the system on which you are running

• Operating system/version: OSX 12.3 (Monterey)
• Computer hardware: M1