caf script should capture MPI scripts with full paths

[jeffhammond]

I installed OpenCoarrays like this:

# MPICH_* override because MPICH was built against GCC-5
export MPICH_CC=gcc-6
export MPICH_FC=gfortran-6
export MPI_DIR=/opt/mpich/dev/gcc/default # Jeff's Mac
git clone https://github.com/sourceryinstitute/opencoarrays.git && cd opencoarrays && \
mkdir build && cd build && \
CC=$MPI_DIR/bin/mpicc FC=$MPI_DIR/bin/mpifort cmake .. \
-DCMAKE_INSTALL_PREFIX=/opt/opencoarrays/mpich-dev-gcc-6 \
-DMPI_C_COMPILER=$MPI_DIR/bin/mpicc -DMPI_Fortran_COMPILER=$MPI_DIR/bin/mpifort && \
make -j4 && \
ctest && \
make install

When I went to use caf, I saw this error:

github-official jrhammon$ make allfortran
cd FORTRAN/Synch_p2p;       /Applications/Xcode.app/Contents/Developer/usr/bin/make p2p
/opt/opencoarrays/mpich-dev-gcc-6/bin/caf -std=f2008 -ffree-form -cpp -O3  -DPRKVERSION="'2.16'" -DRADIUS=  p2p.f90 -o p2p
ifort: command line warning #10006: ignoring unknown option '-std=f2008'
ifort: command line warning #10006: ignoring unknown option '-ffree-form'
ifort: command line warning #10006: ignoring unknown option '-fcoarray=lib'
^Cmake[1]: *** [p2p] Error 2
make: *** [allfortranserial] Interrupt: 2

This is because I have mpif90 associated with ifort in my PATH. I rely upon absolute paths to get GCC and LLVM MPI wrappers, as I did above.

Unfortunately, caf doesn't remember what MPI wrapper was used to build the library:

# Default to "mpif90" Fortran compiler if environment variable CAFC has zero length:
if [ -z "$CAFC" ]; then
   CAFC=mpif90
fi

It really should do this, both because of people like me who don't necessarily have the right mpif90 first in their PATH on purpose, but also for anyone that doesn't know better and switches MPI installation.

This will likely prevent all sorts of awful bug reports associated with people that compile with MPICH but try to run with Open-MPI, and vice versa.

[zbeekman]

I agree, and this has been on my to-do list for quite some time. I'll dig into it on Wednesday or Thursday

[rouson]

@jefhammond, great catch. Thanks for all the fabulous feedback. Keep it coming.

A quick fix is to insert a line of the following form at the top of the caf script:

source <opencoarrays-install-dir>/setup.sh

which will prepend absolute paths in the PATH environment variable for the process executing caf. The setup.sh file (or setup.csh for C-shells) contains source statements for all OpenCoarrays-installed software. I think it correctly reflects any pre-installed software the user specified with in the way you did by your setting CC and FC. I'll verify this and make sure user choices are respected. I'll aim to have this done shortly. I'll see if we can get a bug fix release out this week that reflects some or all of your feedback. If so, it would be great if you could test it for us.

[jeffhammond]

I just manually hacked caf with the absolute path in my local install, and Travis is still blocked on stupid CMake BS that I do not see locally, so this isn't an urgent issue for me. However, I will be happy to test.

[zbeekman]

@jeffhammond I could look at the CMake problems if you think I could be helpful... Is it related to OpenCoarrays or other packages?

[jeffhammond]

@zbeekman CMake is just very stupid about detecting MPI relative to Autotools. I think I fixed it, it just required 3x more work than is reasonable.

[zbeekman]

@jeffhammond OK. Just curious, are you using the find_package capability?

Yes CMake is not designed so well... but I have yet to find an alternative that I like more. I'd love to check out Meson at some point and see if it's a bit saner, but am nervous about its relative new-ness and stability.

[jeffhammond]

@zbeekman

I am just invoking CMake with CC=$MPI_PATH/bin/mpicc and foolishly expecting that CMake can figure out that the libraries it should use are in $MPI_PATH/lib instead of some other location.

I cannot prove Autotools is broken, so I have not tried to fix it. And yes, not having Windows support is a feature of Autotools in my book 😉

[rouson]

My earlier suggestion was based on a mistaken assumption that our install.sh script launched the build. I realized a little while ago that the scenario described in this issue involves a user launching a build by directly invoking CMake. The previously proposed solution would not have worked because the aforementioned setup.sh file is written by install.sh.

Pull request #197 fixes this issue by directly modifying the CMake script that builds the "caf" compilation script. Hence, the fix should work whether a user installs via our installation script or simply via CMake directly.

I'm closing this issue. We can reopen it if #197 fails.

[zbeekman]

@rouson If you say "fixes #193" in the merge commit comments (possibly in the pull request description as well) it will automatically close this issue once it gets merged, just FYI. closing issues with commit messages

[zbeekman]

@jeffhammond I believe this issue is resolved in the latest release and the code in master. Please let me know if you find otherwise.

[jeffhammond]

Thanks. Will test next time I do a local build of PRK.