Travis CI tests failing due to apparent lack of --oversubscribe

[jeffhammond]

Avg response time

Please make sure to put any logs, terminal output, or code in
[fenced code blocks]. Please also read the [contributing guidelines]
before submitting a new issue.

Defect/Bug Report

The title of the issue should start with Defect: followed by a
succinct title.

• OpenCoarrays Version: opencoarrays-1.9.2_1.el_capitan.bottle.tar.gz
• Fortran Compiler: gfortran-7
• C compiler used for building lib: gcc but it doesn't matter
• Installation method: Homebrew
• Output of uname -a: x86_64-apple-darwin15.6.0
• MPI library being used: open-mpi-3.0.0.el_capitan.bottle.tar.gz
• Machine architecture and number of physical cores: AWS 1.5 vcores (Travis CI)
• Version of CMake: not relevant because I'm using Homebrew

Full details available from https://travis-ci.org/ParRes/Kernels/jobs/297391851.

Observed Behavior

+cafrun -n 4 FORTRAN/p2p-coarray 10 1024 1024
--------------------------------------------------------------------------
There are not enough slots available in the system to satisfy the 4 slots
that were requested by the application:
  FORTRAN/p2p-coarray
Either request fewer slots for your application, or make more slots available
for use.
--------------------------------------------------------------------------

Expected Behavior

Program runs normally.

Steps to Reproduce

Run in Travis CI with 4 images on 1.5 vcores or any other scenario that oversubscribes hardware threads.

Details

open-mpi/ompi#3133 and https://github.com/sourceryinstitute/OpenCoarrays/blob/8e049dc3d9a8824ebfb6290cc301ac8cb98cdce5/CMakeLists.txt indicate this should not happen, so I am confused.

[zbeekman]

open-mpi/ompi#3133 and /CMakeLists.txt@8e049dc indicate this should not happen, so I am confused.

Well, as far as the CMake logic goes, we check when you run CTest or make check whether or not we need to pass --oversubscribe, but how are we supposed to know at runtime if your job needs to be oversubscribed? Homebrew is the perfect case and point because the binaries are built on a different machine entirely.

Suggested work around

You can just pass --oversubscribe as an argument to cafrun and it should get passed through to mpiexec. Please let me know if this isn't working or if you have any questions about this. You can add the -s or --show flag to print a dry run of what cafrun will invoke. Note that I need to update cafrun's --help text which does not discuss this option, but it is shown in the examples and is present in the manpages.

discussion of the issue/feature request

OK---to be fair---I'm being a little bit ridiculous, because usually there is some way to query at runtime how many cores there are, and OpenMPI is a popular enough MPI implementation that it would not be completely unreasonable to detect it at configure/CMake time, embed in cafrun the possibility of some runtime diagnostics to determine whether --oversubscribe needs to be passed through to the OpenMPI runtime launcher.

So, honest question, @jeffhammond (and @rouson): Do you think that cafrun should be trying to handle whether or not --oversubscribe gets passed?

I think the closest thing we could do is enable automatic oversubscription in the event that OpenCoarrays was built against OMPI AND we're currently running on macOS, since it's unlikely that it is a macOS machine that is not a simple shared memory machine.

I really don't know, in general (think a batch job on a Cray or SGI machine), how our launcher script will be able to query the number of MPI ranks available to the process to that it is about to run. For now I'm going to mark this as a request for enhancement, and "won't fix" since I don't see a reasonable way of being able to implement this in a generic manner. If you think a macOS+OMPI only solution seems worthwhile then I'll consider implementing it.

[jeffhammond]

I'll just use cafrun --oversubscribe in my Travis scripts.

If I was going to try to maximize user experience, I'd detect Travis CI in cafrun but that's a bit of a hack.

[zbeekman]

I like the Travis-CI and macOS detection combo. Travis is always going to be excessively constrained core-wise.

…

On Mon, Nov 6, 2017 at 2:43 PM Jeff Hammond ***@***.***> wrote: Closed #460 <#460> . — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#460 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAREPNsjezlyI5B4VuyfcXWS3pkA48Xaks5sz2F6gaJpZM4QSbWa> .

[jeffhammond]

@zbeekman FYI for some reason it does not work...

+make -C FORTRAN coarray
/usr/local/bin/caf -std=f2008 -cpp  -DRADIUS=2 -DSTAR p2p-coarray.f90 -fcoarray=single -o p2p-coarray
/usr/local/bin/caf -std=f2008 -cpp  -DRADIUS=2 -DSTAR stencil-coarray.f90 -fcoarray=single -o stencil-coarray
/usr/local/bin/caf -std=f2008 -cpp  -DRADIUS=2 -DSTAR transpose-coarray.f90 -fcoarray=single -o transpose-coarray
+export PRK_MPI_PROCS=4
+PRK_MPI_PROCS=4
+'[' gcc = gcc ']'
+'[' osx = osx ']'
+export 'PRK_LAUNCHER=/usr/local/bin/cafrun --oversubscribe'
+PRK_LAUNCHER='/usr/local/bin/cafrun --oversubscribe'
+export TMPDIR=/tmp
+TMPDIR=/tmp
+/usr/local/bin/cafrun --oversubscribe -n 4 FORTRAN/p2p-coarray 10 1024 1024
+/usr/local/bin/cafrun --oversubscribe -n 4 FORTRAN/stencil-coarray 10 1000
+/usr/local/bin/cafrun --oversubscribe -n 4 FORTRAN/transpose-coarray 10 1024 1
+/usr/local/bin/cafrun --oversubscribe -n 4 FORTRAN/transpose-coarray 10 1024 32
The command "sh ./travis/build-run-prk.sh $TRAVIS_ROOT $PRK_TARGET" exited with 0.
Done. Your build exited with 0.
/Users/travis/.travis/job_stages: line 166: shell_session_update: command not found

https://travis-ci.org/ParRes/Kernels/jobs/300815486

[zbeekman]

Hi Jeff, are you at SC? I'm in the PGAS workshop now and it would be great to meet you in person.

RE the issue: I'm sorry for your troubles; this wrapper script still needs some serious love. The issue is that -np (or -n) followed by the number of images needs to be the first and second arguments to cafrun. Yes, I know this is super lame. I hope to have some time to clean this up, but things have been nuts the past few weeks. If you move --oversubscribeafter-n 4` it should work.

[jeffhammond]

Hi Jeff, are you at SC? I'm in the PGAS workshop now and it would be great to meet you in person.

No, sorry. I am on parental leave October and November.

RE the issue: I'm sorry for your troubles; this wrapper script still needs some serious love. The issue is that -np (or -n) followed by the number of images needs to be the first and second arguments tocafrun. Yes, I know this is super lame.

I’m glad you said it first 😉

I hope to have some time to clean this up, but things have been nuts the past few weeks. If you move--oversubscribeafter-n 4` it should work.

Will try this shortly. Or approve your PR 😁