Stuck on the last test of each file

[oleiade]

Hi,

I'm extensively using the bats testing framework in trousseau. I have written a bunch of tests. Everything works fine on my laptop (OSX), and in vagrant boxes (Ubuntu 14.04), however when I run the tests in a CI environment like codeship or travis I noticed that bats always end up stuck on the last test of each files.

For example:

~/bats/bin/bats -t tests/create.bats
1..6
ok 1 create store with one valid recipient succeeds
ok 2 create symmetric store succeeds
ok 3 create generates a file in 0600 mode
ok 4 create store with one invalid recipient fails
ok 5 create store without a recipient fails
ok 6 create store with one valid recipient and one invalid recipient fails
# Hangs here, forever

Any ideas what could cause this? a wait4 syscall never ending? A SIGCHLD never received? An improper redirection of stdout/stderr hiding the real problem?

Thanks for your help :-)
Theo

[Sylvain303]

Could you isolate the failing test in a short example?

I got some hangs too, not reproducible yet, it may be related to IFS see #89.

could you give your bash version?

[oleiade]

Unfortunately, I can't isolate any specific test failing. The only thing common to all the hang ups is that they always happen on the last test of the first test file evaluated.

After I read it, I'm not so sure it is related to #89 (I actually don't know what this IFS is neither)

[Sylvain303]

$IFS is internal bash variable, used in auto spliting, I've pushed on my forked version and preparing the pull request.

from man bash:

IFS    The  Internal  Field  Separator  that is used for word splitting
       after expansion and to split lines  into  words  with  the  read 
       builtin  command.   The  default  value  is ``<space><tab><newine>''.

[oleiade]

Thanks for the info :-)

Gonna try with your fork see if it fixes things, never know...

[filex]

I might have the same issue: In one of my tests I call an /etc/init.d script. As long as I do this, bats doesn't return after the last check.

This is a simple way to reproduce it:

#!/usr/bin/env bats

@test "fork something" {
        sleep 5 &
        echo 'forked'
}

@test "hanging check" {
        echo 'hang'
}

This stalls bats until the 5s are over. This version may come closer to my init-script case:

#!/usr/bin/env bats

@test "fork something" {
        (while true; do sleep 1; done) &
        echo 'forked'
}

@test "hanging check" {
        echo 'hang'
}

You have to kill the wait with ctrl-c, then the check mark appears next to "hanging check" and bats terminates.

I see the same behaviour in master-branch and in tags/v0.4.0.

I'm new to bats. thus is blind guessing: Does bats look after forked processes and waits for them to terminate?

[filex]

In my init-script example I can terminate the hang by stopping the started daemon from another shell.

Furthermore, I have found a race condition in my test. One check has started the Apache httpd and another was supposed to stop it. Without anything else to test and do between start and stop, I believe the stop-test to be triggered before the forked httpd was ready to accept shutdown signals.

With more tests and some tactical sleeps I can make sure the forked processes are stopped before the end of the test suite. Then, bats terminates as expected.

@oleiade, do you have any "asymmetric" forks in your test, too?

[Sylvain303]

@filex why not waiting for the pid? maybe starting the process with nohup or something similar writing the pid into a file. You don't start a test in a one test, and check the result in another, aren't you?

http://stackoverflow.com/questions/356100/how-to-wait-in-bash-for-several-subprocesses-to-finish-and-return-exit-code-0

I think you have to detach the process looping forever in order do finish the test.

[filex]

Yes, @Sylvain303, I start a daemon, run several checks and stop it lastly. This is split up in several tests to have somewhat meaningful test names for everything. My problem was the race condition between start and stop. Everything is working fine now.

But apparently bats does some kind of waiting itself. Do you know how and why?

When I call an init script (apache and my own software) in a shell, the processes are forked and vanish from the shells jobs list. Thus waiting does not work and returns immediately.

However, bats manages to wait until the forked processes are shut down. I wouldn't exactly call that a bug, but the behaviour was unexepected.

[Sylvain303]

hum, no idea for the moment. I stepped in bats code for a bug, it parses the .bats, generate a valid shell script in a tmp and execute each function, I gonna try your bug to see I can see where it hangs. I will probably to have to write similar tests. But dont expect a fast reply from me ;)…

[myoung34]

I ran into this and tried nohup, &, pid catching, all of those.
The only way to solve it was to write an init.d script to do it for me....

[esiegerman]

It's not stuck on the last test, but after it...

The problem is that bats uses file descriptor 3 as a communication channel among its internal processes. Your background sleep process inherits FD#3 from bats, as the write end of a pipe. Since your process is holding FD#3 open, the pipe doesn't report EOF to its reader, and bats, which is waiting for that to happen, hangs. When the sleep process exits, its FD#3 gets closed, and since that was the last writer, the pipe finally reports EOF.

I've tried to come up with a patch for bats, but haven't suceeded. I'd have expected this to work:

diff --git a/libexec/bats-exec-test b/libexec/bats-exec-test
index 044b2c3..0ef541f 100755
--- a/libexec/bats-exec-test
+++ b/libexec/bats-exec-test
@@ -321,7 +321,7 @@ bats_perform_test() {
     trap "bats_debug_trap \"\$BASH_SOURCE\"" debug
     trap "bats_error_trap" err
     trap "bats_teardown_trap" exit
-    "$BATS_TEST_NAME" >>"$BATS_OUT" 2>&1
+    "$BATS_TEST_NAME" >>"$BATS_OUT" 2>&1 3>-
     BATS_TEST_COMPLETED=1

   else

but it doesn't; with it, "pretty output" only contains the checkmarks, no test names! I don't know why.

As a workaround, try changing the sleep in your test to:

sleep 5 3>- &

[Dentrax]

@esiegerman Thanks for your workaround. Can you please explain what 3>- does? I could not find this on Google, this usage is the first time I've seen. Looks interesting!