Test: coarray_navier_stokes fails

[finneyb]

On 12/6/2015 10:16 AM, afanfa wrote:

The problem with the navier_stokes solver is most likely related with
the FFTs and WALLTIME functions written for x86. You may want to try
with LEGACY_ARCHITECTURE=ON and HIGH_RESOLUTION_TIMER=OFF ("Advanced
options" in INSTALL.md).

I added the suggested options at the appropriate place in install.sh and then executed install. Running ctest again results in the same error. The directory pde_solvers/navier-stokes is present, but the directory does not contain a file or directory named coarray_navier_stokes

22/22 Testing: coarray_navier_stokes
22/22 Test: coarray_navier_stokes
Command: "/usr/local/bin/mpich/bin/mpiexec" "-np" "2" "/usr/local/bin/opencoarrays/opencoarrays-build/src/tests/integration/pde_solvers/navier-stokes/coarray_navier_stokes"
Directory: /usr/local/bin/opencoarrays/opencoarrays-build
"coarray_navier_stokes" start time: Dec 06 21:03 PST

Output:

[proxy:0:0@madison] HYDU_create_process (/usr/local/bin/mpich-3.1.4/src/pm/hydra/utils/launch/launch.c:75): execvp error on file /usr/local/bin/opencoarrays/opencoarrays-build/src/tests/integration/pde_solvers/navier-stokes/coarray_navier_stokes (No such file or directory)
[proxy:0:0@madison] HYDU_create_process (/usr/local/bin/mpich-3.1.4/src/pm/hydra/utils/launch/launch.c:75): execvp error on file /usr/local/bin/opencoarrays/opencoarrays-build/src/tests/integration/pde_solvers/navier-stokes/coarray_navier_stokes (No such file or directory)

= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= PID 26466 RUNNING AT madison
= EXIT CODE: 255
= CLEANING UP REMAINING PROCESSES

= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES

## Test time = 0.03 sec

Test Fail Reason:
Required regular expression not found.Regex=[Test passed.
]
"coarray_navier_stokes" end time: Dec 06 21:03 PST

"coarray_navier_stokes" time elapsed: 00:00:00

Thanks for submitting this issue. If that is the only test failing, then it is very likely that your build is correct. I'll therefore assume this is not a critical issue for you and it will be several days before I have a chance to investigate this further.

[zbeekman]

I am able to recreate the same behavior on a Travis-CI Ubuntu linux environment. Also, it seems that this test isn't even attempted under OS X. Just curious what the reason is for this.

It's not attempted on OS X because it failed on OS X due to an incompatibility with the FFT routines. As mentioned in the other thread, I'd be glad to switch to other FFTs as the default if it's something you'd like to try.

[zbeekman]

@finneyb: Could you please re-run ctest and add the --verbose flag? (i.e. ctest --verbose) and also paste the output into a fenced code block so that it doesn't get interpreted as markdown?

 ```
 < *code* >
 ```

@afanfa would you mind taking a quick look at this error message on the Travis-CI build? I'm guessing this is due to an issue with the FFT library, but I'm not sure... I was reading that some people had problems with /dev/urandom on Travis-CI as well, but according to my testing that is not the issue. Here is the log line where the error message starts: https://travis-ci.org/zbeekman/opencoarrays/jobs/96362389#L1301

@afanfa & @rouson my testing on Travis-CI also seems to indicate that libfft_sse.a is always used for linking no matter what -DLEGACY_ARCHITECTURE= is set to... I'm can try to investigate more, but for now this is challenging, since I can only build locally on a Mac.

[finneyb]

sorry, forgot the code block. Here it is again.

Start testing: Dec 11 16:03 PST
----------------------------------------------------------
1/22 Testing: initialize_mpi
1/22 Test: initialize_mpi
Command: "/usr/local/bin/mpich/bin/mpiexec" "-np" "2" "/usr/local/bin/opencoarrays/opencoarrays-build/src/tests/unit/init_register/initialize_mpi"
Directory: /usr/local/bin/opencoarrays/opencoarrays-build
"initialize_mpi" start time: Dec 11 16:03 PST
Output:
----------------------------------------------------------
 Test passed.
<end of output>
Test time =   0.05 sec
----------------------------------------------------------
Test Pass Reason:
Required regular expression found.Regex=[Test passed.
]
"initialize_mpi" end time: Dec 11 16:03 PST
"initialize_mpi" time elapsed: 00:00:00
----------------------------------------------------------

2/22 Testing: register
2/22 Test: register
Command: "/usr/local/bin/mpich/bin/mpiexec" "-np" "2" "/usr/local/bin/opencoarrays/opencoarrays-build/src/tests/unit/init_register/register"
Directory: /usr/local/bin/opencoarrays/opencoarrays-build
"register" start time: Dec 11 16:03 PST
Output:
----------------------------------------------------------
 Test passed.
<end of output>
Test time =   0.04 sec
----------------------------------------------------------
Test Pass Reason:
Required regular expression found.Regex=[Test passed.
]
"register" end time: Dec 11 16:03 PST
"register" time elapsed: 00:00:00
----------------------------------------------------------

3/22 Testing: register_rename_me
3/22 Test: register_rename_me
Command: "/usr/local/bin/mpich/bin/mpiexec" "-np" "2" "/usr/local/bin/opencoarrays/opencoarrays-build/src/tests/unit/init_register/register_rename_me"
Directory: /usr/local/bin/opencoarrays/opencoarrays-build
"register_rename_me" start time: Dec 11 16:03 PST
Output:
----------------------------------------------------------
 Test passed.
<end of output>
Test time =   0.04 sec
----------------------------------------------------------
Test Pass Reason:
Required regular expression found.Regex=[Test passed.
]
"register_rename_me" end time: Dec 11 16:03 PST
"register_rename_me" time elapsed: 00:00:00
----------------------------------------------------------

4/22 Testing: register_rename_me_too
4/22 Test: register_rename_me_too
Command: "/usr/local/bin/mpich/bin/mpiexec" "-np" "2" "/usr/local/bin/opencoarrays/opencoarrays-build/src/tests/unit/init_register/register_rename_me_too"
Directory: /usr/local/bin/opencoarrays/opencoarrays-build
"register_rename_me_too" start time: Dec 11 16:03 PST
Output:
----------------------------------------------------------
 Test passed.
<end of output>
Test time =   0.04 sec
----------------------------------------------------------
Test Pass Reason:
Required regular expression found.Regex=[Test passed.
]
"register_rename_me_too" end time: Dec 11 16:03 PST
"register_rename_me_too" time elapsed: 00:00:00
----------------------------------------------------------

5/22 Testing: allocate_as_barrier
5/22 Test: allocate_as_barrier
Command: "/usr/local/bin/mpich/bin/mpiexec" "-np" "2" "/usr/local/bin/opencoarrays/opencoarrays-build/src/tests/unit/init_register/allocate_as_barrier"
Directory: /usr/local/bin/opencoarrays/opencoarrays-build
"allocate_as_barrier" start time: Dec 11 16:03 PST
Output:
----------------------------------------------------------
 Test passed.
<end of output>
Test time =   1.04 sec
----------------------------------------------------------
Test Pass Reason:
Required regular expression found.Regex=[Test passed.
]
"allocate_as_barrier" end time: Dec 11 16:03 PST
"allocate_as_barrier" time elapsed: 00:00:01
----------------------------------------------------------

6/22 Testing: allocate_as_barrier_proc
6/22 Test: allocate_as_barrier_proc
Command: "/usr/local/bin/mpich/bin/mpiexec" "-np" "2" "/usr/local/bin/opencoarrays/opencoarrays-build/src/tests/unit/init_register/allocate_as_barrier_proc"
Directory: /usr/local/bin/opencoarrays/opencoarrays-build
"allocate_as_barrier_proc" start time: Dec 11 16:03 PST
Output:
----------------------------------------------------------
 Test passed.
<end of output>
Test time =   1.04 sec
----------------------------------------------------------
Test Pass Reason:
Required regular expression found.Regex=[Test passed.
]
"allocate_as_barrier_proc" end time: Dec 11 16:03 PST
"allocate_as_barrier_proc" time elapsed: 00:00:01
----------------------------------------------------------

7/22 Testing: get_array
7/22 Test: get_array
Command: "/usr/local/bin/mpich/bin/mpiexec" "-np" "2" "/usr/local/bin/opencoarrays/opencoarrays-build/src/tests/unit/send-get/get_array"
Directory: /usr/local/bin/opencoarrays/opencoarrays-build
"get_array" start time: Dec 11 16:03 PST
Output:
----------------------------------------------------------
 Test passed
 Test passed
<end of output>
Test time =   0.64 sec
----------------------------------------------------------
Test Pass Reason:
Required regular expression found.Regex=[Test passed.
]
"get_array" end time: Dec 11 16:03 PST
"get_array" time elapsed: 00:00:00
----------------------------------------------------------

8/22 Testing: send_array
8/22 Test: send_array
Command: "/usr/local/bin/mpich/bin/mpiexec" "-np" "2" "/usr/local/bin/opencoarrays/opencoarrays-build/src/tests/unit/send-get/send_array"
Directory: /usr/local/bin/opencoarrays/opencoarrays-build
"send_array" start time: Dec 11 16:03 PST
Output:
----------------------------------------------------------
 Test passed
 Test passed
<end of output>
Test time =   0.88 sec
----------------------------------------------------------
Test Pass Reason:
Required regular expression found.Regex=[Test passed.
]
"send_array" end time: Dec 11 16:03 PST
"send_array" time elapsed: 00:00:00
----------------------------------------------------------

9/22 Testing: get_with_offset_1d
9/22 Test: get_with_offset_1d
Command: "/usr/local/bin/mpich/bin/mpiexec" "-np" "2" "/usr/local/bin/opencoarrays/opencoarrays-build/src/tests/unit/send-get/get_with_offset_1d"
Directory: /usr/local/bin/opencoarrays/opencoarrays-build
"get_with_offset_1d" start time: Dec 11 16:03 PST
Output:
----------------------------------------------------------
 Test passed.
<end of output>
Test time =   0.04 sec
----------------------------------------------------------
Test Pass Reason:
Required regular expression found.Regex=[Test passed.
]
"get_with_offset_1d" end time: Dec 11 16:03 PST
"get_with_offset_1d" time elapsed: 00:00:00
----------------------------------------------------------

10/22 Testing: whole_get_array
10/22 Test: whole_get_array
Command: "/usr/local/bin/mpich/bin/mpiexec" "-np" "2" "/usr/local/bin/opencoarrays/opencoarrays-build/src/tests/unit/send-get/whole_get_array"
Directory: /usr/local/bin/opencoarrays/opencoarrays-build
"whole_get_array" start time: Dec 11 16:03 PST
Output:
----------------------------------------------------------
 Test passed.
<end of output>
Test time =   0.05 sec
----------------------------------------------------------
Test Pass Reason:
Required regular expression found.Regex=[Test passed.
]
"whole_get_array" end time: Dec 11 16:03 PST
"whole_get_array" time elapsed: 00:00:00
----------------------------------------------------------

11/22 Testing: strided_get
11/22 Test: strided_get
Command: "/usr/local/bin/mpich/bin/mpiexec" "-np" "2" "/usr/local/bin/opencoarrays/opencoarrays-build/src/tests/unit/send-get/strided_get"
Directory: /usr/local/bin/opencoarrays/opencoarrays-build
"strided_get" start time: Dec 11 16:03 PST
Output:
----------------------------------------------------------
 Test passed.
<end of output>
Test time =   0.05 sec
----------------------------------------------------------
Test Pass Reason:
Required regular expression found.Regex=[Test passed.
]
"strided_get" end time: Dec 11 16:03 PST
"strided_get" time elapsed: 00:00:00
----------------------------------------------------------

12/22 Testing: co_sum
12/22 Test: co_sum
Command: "/usr/local/bin/mpich/bin/mpiexec" "-np" "4" "/usr/local/bin/opencoarrays/opencoarrays-build/src/tests/unit/collectives/co_sum_test"
Directory: /usr/local/bin/opencoarrays/opencoarrays-build
"co_sum" start time: Dec 11 16:03 PST
Output:
----------------------------------------------------------
 Test passed.
<end of output>
Test time =   0.13 sec
----------------------------------------------------------
Test Pass Reason:
Required regular expression found.Regex=[Test passed.
]
"co_sum" end time: Dec 11 16:03 PST
"co_sum" time elapsed: 00:00:00
----------------------------------------------------------

13/22 Testing: co_broadcast
13/22 Test: co_broadcast
Command: "/usr/local/bin/mpich/bin/mpiexec" "-np" "4" "/usr/local/bin/opencoarrays/opencoarrays-build/src/tests/unit/collectives/co_broadcast_test"
Directory: /usr/local/bin/opencoarrays/opencoarrays-build
"co_broadcast" start time: Dec 11 16:03 PST
Output:
----------------------------------------------------------
 Test passed.
<end of output>
Test time =   0.10 sec
----------------------------------------------------------
Test Pass Reason:
Required regular expression found.Regex=[Test passed.
]
"co_broadcast" end time: Dec 11 16:03 PST
"co_broadcast" time elapsed: 00:00:00
----------------------------------------------------------

14/22 Testing: co_min
14/22 Test: co_min
Command: "/usr/local/bin/mpich/bin/mpiexec" "-np" "4" "/usr/local/bin/opencoarrays/opencoarrays-build/src/tests/unit/collectives/co_min_test"
Directory: /usr/local/bin/opencoarrays/opencoarrays-build
"co_min" start time: Dec 11 16:03 PST
Output:
----------------------------------------------------------
 Test passed.
<end of output>
Test time =   0.10 sec
----------------------------------------------------------
Test Pass Reason:
Required regular expression found.Regex=[Test passed.
]
"co_min" end time: Dec 11 16:03 PST
"co_min" time elapsed: 00:00:00
----------------------------------------------------------

15/22 Testing: co_max
15/22 Test: co_max
Command: "/usr/local/bin/mpich/bin/mpiexec" "-np" "4" "/usr/local/bin/opencoarrays/opencoarrays-build/src/tests/unit/collectives/co_max_test"
Directory: /usr/local/bin/opencoarrays/opencoarrays-build
"co_max" start time: Dec 11 16:03 PST
Output:
----------------------------------------------------------
 Correct real(c_double) co_max
 Test passed.
<end of output>
Test time =   0.12 sec
----------------------------------------------------------
Test Pass Reason:
Required regular expression found.Regex=[Test passed.
]
"co_max" end time: Dec 11 16:03 PST
"co_max" time elapsed: 00:00:00
----------------------------------------------------------

16/22 Testing: syncall
16/22 Test: syncall
Command: "/usr/local/bin/mpich/bin/mpiexec" "-np" "32" "/usr/local/bin/opencoarrays/opencoarrays-build/src/tests/unit/sync/syncall"
Directory: /usr/local/bin/opencoarrays/opencoarrays-build
"syncall" start time: Dec 11 16:03 PST
Output:
----------------------------------------------------------
 Test passed.
<end of output>
Test time =   7.31 sec
----------------------------------------------------------
Test Pass Reason:
Required regular expression found.Regex=[Test passed.
]
"syncall" end time: Dec 11 16:03 PST
"syncall" time elapsed: 00:00:07
----------------------------------------------------------

17/22 Testing: syncimages
17/22 Test: syncimages
Command: "/usr/local/bin/mpich/bin/mpiexec" "-np" "32" "/usr/local/bin/opencoarrays/opencoarrays-build/src/tests/unit/sync/syncimages"
Directory: /usr/local/bin/opencoarrays/opencoarrays-build
"syncimages" start time: Dec 11 16:03 PST
Output:
----------------------------------------------------------
 Test passed.
<end of output>
Test time =   8.33 sec
----------------------------------------------------------
Test Pass Reason:
Required regular expression found.Regex=[Test passed.
]
"syncimages" end time: Dec 11 16:04 PST
"syncimages" time elapsed: 00:00:08
----------------------------------------------------------

18/22 Testing: co_reduce
18/22 Test: co_reduce
Command: "/usr/local/bin/mpich/bin/mpiexec" "-np" "4" "/usr/local/bin/opencoarrays/opencoarrays-build/src/tests/unit/collectives/co_reduce_test"
Directory: /usr/local/bin/opencoarrays/opencoarrays-build
"co_reduce" start time: Dec 11 16:04 PST
Output:
----------------------------------------------------------
 Test passed.
<end of output>
Test time =   0.15 sec
----------------------------------------------------------
Test Pass Reason:
Required regular expression found.Regex=[Test passed.
]
"co_reduce" end time: Dec 11 16:04 PST
"co_reduce" time elapsed: 00:00:00
----------------------------------------------------------

19/22 Testing: hello_multiverse
19/22 Test: hello_multiverse
Command: "/usr/local/bin/mpich/bin/mpiexec" "-np" "2" "/usr/local/bin/opencoarrays/opencoarrays-build/src/tests/integration/coarrayHelloWorld/hello_multiverse"
Directory: /usr/local/bin/opencoarrays/opencoarrays-build
"hello_multiverse" start time: Dec 11 16:04 PST
Output:
----------------------------------------------------------
 Greetings from image  1 of  2                                                                       
 Greetings from image  2 of  2                                                                       
 Test passed.
<end of output>
Test time =   0.04 sec
----------------------------------------------------------
Test Pass Reason:
Required regular expression found.Regex=[Test passed.
]
"hello_multiverse" end time: Dec 11 16:04 PST
"hello_multiverse" time elapsed: 00:00:00
----------------------------------------------------------

20/22 Testing: coarray_burgers_pde
20/22 Test: coarray_burgers_pde
Command: "/usr/local/bin/mpich/bin/mpiexec" "-np" "2" "/usr/local/bin/opencoarrays/opencoarrays-build/src/tests/integration/pde_solvers/coarrayBurgers/coarray_burgers_pde"
Directory: /usr/local/bin/opencoarrays/opencoarrays-build
"coarray_burgers_pde" start time: Dec 11 16:04 PST
Output:
----------------------------------------------------------
 Time =  0.60065795534754485     
 On image            2 u =   1.9805929082814393E-015  -2.5585980904689749       -3.1602423341809680       -2.7616422370249771       -2.2228610006371614       -1.6695096165091943       -1.1137114029330646      -0.55703975723770482     
 On image            1 u =  -3.6062644769485318E-016  0.55703975723770405        1.1137114029330639        1.6695096165091936        2.2228610006371610        2.7616422370249762        3.1602423341809680        2.5585980904689771     
 Test passed.
<end of output>
Test time =   0.11 sec
----------------------------------------------------------
Test Pass Reason:
Required regular expression found.Regex=[Test passed.
]
"coarray_burgers_pde" end time: Dec 11 16:04 PST
"coarray_burgers_pde" time elapsed: 00:00:00
----------------------------------------------------------

21/22 Testing: co_heat
21/22 Test: co_heat
Command: "/usr/local/bin/mpich/bin/mpiexec" "-np" "2" "/usr/local/bin/opencoarrays/opencoarrays-build/src/tests/integration/pde_solvers/coarrayHeatSimplified/co_heat"
Directory: /usr/local/bin/opencoarrays/opencoarrays-build
"co_heat" start time: Dec 11 16:04 PST
Output:
----------------------------------------------------------
 Test passed.
<end of output>
Test time =  32.56 sec
----------------------------------------------------------
Test Pass Reason:
Required regular expression found.Regex=[Test passed.
]
"co_heat" end time: Dec 11 16:04 PST
"co_heat" time elapsed: 00:00:32
----------------------------------------------------------

22/22 Testing: coarray_navier_stokes
22/22 Test: coarray_navier_stokes
Command: "/usr/local/bin/mpich/bin/mpiexec" "-np" "2" "/usr/local/bin/opencoarrays/opencoarrays-build/src/tests/integration/pde_solvers/navier-stokes/coarray_navier_stokes"
Directory: /usr/local/bin/opencoarrays/opencoarrays-build
"coarray_navier_stokes" start time: Dec 11 16:04 PST
Output:
----------------------------------------------------------
[proxy:0:0@madison] HYDU_create_process (/usr/local/bin/mpich-3.1.4/src/pm/hydra/utils/launch/launch.c:75): execvp error on file /usr/local/bin/opencoarrays/opencoarrays-build/src/tests/integration/pde_solvers/navier-stokes/coarray_navier_stokes (No such file or directory)
[proxy:0:0@madison] HYDU_create_process (/usr/local/bin/mpich-3.1.4/src/pm/hydra/utils/launch/launch.c:75): execvp error on file /usr/local/bin/opencoarrays/opencoarrays-build/src/tests/integration/pde_solvers/navier-stokes/coarray_navier_stokes (No such file or directory)

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 15397 RUNNING AT madison
=   EXIT CODE: 255
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
<end of output>
Test time =   0.03 sec
----------------------------------------------------------
Test Fail Reason:
Required regular expression not found.Regex=[Test passed.
]
"coarray_navier_stokes" end time: Dec 11 16:04 PST
"coarray_navier_stokes" time elapsed: 00:00:00
----------------------------------------------------------

End testing: Dec 11 16:04 PST

[zbeekman]

@finneyb Ah, OK. I should have noticed earlier, you're experiencing a completely different issue than I am...

@finneyb can you please tell me the output of running uname -mps on the machine you're running this on, as well as the command(s) you used to install open coarrays? (e.g. ./install.sh /usr/local/bin or mkdir build && cd build && cmake .. && make && make install etc.)

It looks like either there may be a problem in the logic of how the navier_stokes test is conditionally compiled and added to the list of tests in the CMakeLists.txt. (It links against an FFT library written directly in assembly, and only works on x86_64 Linux, so the CMake build tries to ensure that the OS and architecture will support the library before compiling and linking the test that requires it.)

Some secondary hunches as to where the problem could be are:

• Perhaps the CMake installation forgets to install the navier_stokes test along with the others... that is it gets built but not copied to the install location (it might be worthwhile taking a look at the installation logic and locations)
• An issue using the install.sh as apposed to the "manual" cmake install... I think this is unlikely.

I'll try to look into this some more.

[finneyb]

On 12/11/2015 4:37 PM, Izaak Beekman wrote:

@finneyb https://github.com/finneyb Ah, OK. I should have noticed
earlier, you're experiencing a completely different issue than I am...

@finneyb https://github.com/finneyb can you please tell me the output
of running |uname -mps| on the machine you're running this on, as well
as the command(s) you used to install open coarrays? (e.g. |./install.sh
/usr/local/bin| or |mkdir build && cd build && cmake .. && make && make
install| etc.)

uname -mps
Linux armv7l armv7l

installed open coarrays with
./install.sh /usr/local/bin/opencoarrays

Looks like the installer should just not install that test at all as
this machine is not x86_64.

[zbeekman]

yes I'm a bit confused to why it is being attempted... digging around further.... For now it is certainly safe to ignore that error... the FFT library is not part of OCA just part of the tests... I'm working on fixing this now.

[zbeekman]

I can confirm that this test failure is due to the fact that the top level CMakeLists.txt has unintentionally diverged between the master branch and the 1.x branch (branch-1.0.0) that the releases are based off of. This should be fixed in the next release. (I am working to resolve some unintentional differences soon, as I discussed on the phone with Damian.)

This line is the culprit: https://github.com/sourceryinstitute/opencoarrays/blob/branch-1.0.0/CMakeLists.txt#L137

and it should look like: https://github.com/sourceryinstitute/opencoarrays/blob/master/CMakeLists.txt#L174

[naveen-rn]

@zbeekman Hi, I am sorry...I couldn't understand what exactly is this issue with the navier_stokes test. I suppose we fixed the issue involving 32-bit vs 64-bit version build. This test was supposed to run on only the 64-bit version Linux build and should be skipped for others.
Does the previous patch not working on OS X?