Test hangs on OSX 10.9.5 test-domain-no-error-handler-abort-on-uncaught.js

[chexmixbbq]

• Version: v8.0.0-pre
• Platform: Darwin PA013136.local 13.4.0 Darwin Kernel Version 13.4.0: Mon Jan 11 18:17:34 PST 2016; root:xnu-2422.115.15~1/RELEASE_X86_64 x86_64
• Subsystem: test

This test hangs on:

=== release test-domain-no-error-handler-abort-on-uncaught ===
Path: parallel/test-domain-no-error-handler-abort-on-uncaught
Command: out/Release/node /Users/youngj/JsProjects/node/test/parallel/test-domain-no-error-handler-abort-on-uncaught.js

[Trott]

Thanks for reporting the issue. I am unable to replicate this on Darwin Kernel Version 16.1.0 and I have not seen it on the test machines. I'm not sure what could be causing it. Hopefully someone else will have more insight. Please do comment here if you manage to get it working.

[joyeecheung]

I've run into this a few times today too, and I can reproduce this one when running out/Release/node test/parallel/test-domain-no-error-handler-abort-on-uncaught.js on my mac(Darwin Kernel Version 15.6.0).

I got "Illegal instruction: 4" when trying master, v4.4.7 and v6.9.1 to run individual test cases(passing child 1 or other available textIndex) earlier, but currently I can run the individual cases with expected results, although the test script spawning multiple processes for each test case still hangs.

[joyeecheung]

Turns out I can finish the test, it just takes a long time(long enough to fail the test because of timeout when running with the python script).

real	0m16.144s
user	0m0.981s
sys	0m0.330s

another one

real	0m26.977s
user	0m0.960s
sys	0m0.339s

and another one

real	0m57.213s
user	0m0.988s
sys	0m0.359s

And I notice that there are two test/fixtures/child-process-persistent.js still running after the test finishes, that's odd.

[Trott]

Looks like this test launches 13 child processes at once. Can one or both of you see what happens if you only have the first six items in the array and remove the last seven? And then again but with the last seven in the array and the first six removed? If both of those still exhibit problems, how about if there's just one item in the array?

[Trott]

Just for clarity: The array I'm talking about is tests defined starting on line 14. Each item in it is a function that spans several lines. There are 13 functions in the array.

[joyeecheung]

I have tried some combinations, it looks like whenever I hit the else branch(the one launching child processes from the array), the test will run for a long time. I have tried all the individual cases with child arguments(thus hitting the if branch), and they are all fine. Only spawning child processes will make it run for a long time, and all the combinations of test cases I've tried do that(first half, last half, first one, second one, ...).

BTW aren't there 10 functions in that array? That's what I get from

 if (process.argv[2] === 'child') {
   const testIndex = +process.argv[3];
+  console.log('running test ' + testIndex);
   tests[testIndex]();
 } else {
-
   tests.forEach(function(test, testIndex) {
     var testCmd = '';
     if (!common.isWindows) {
       // Do not create core files, as it can take a lot of disk space on
@@ -156,6 +175,7 @@ if (process.argv[2] === 'child') {
     testCmd += ' ' + 'child';
     testCmd += ' ' + testIndex;

+    console.log('spawn child ' + testIndex);
     var child = child_process.exec(testCmd);

     child.on('exit', function onExit(exitCode, signal) {
@@ -164,5 +184,9 @@ if (process.argv[2] === 'child') {
                      ' and signal ' + signal;
       assert(common.nodeProcessAborted(exitCode, signal), errMsg);
     });
+
+    child.stdout.on('data', function(data) {
+      console.log(data);
+    });
   });
 }

I've tried adding some logs in the tests and see what they would do. All test cases can run to the point before they throw an error out.

[joyeecheung]

Update:

I tried to log in the onexit callback. Most of the times half of the test cases hit that line and log immediately, and after dozens of seconds, another half hit there and log and then quit. It's pretty random which ones will be in the first batch, which ones the second, and it's not ordered. Is there a race or anything?

@@ -156,13 +175,19 @@ if (process.argv[2] === 'child') {
     testCmd += ' ' + 'child';
     testCmd += ' ' + testIndex;

     var child = child_process.exec(testCmd);

     child.on('exit', function onExit(exitCode, signal) {
+      console.log(exitCode, signal, testIndex);
       const errMsg = 'Test at index ' + testIndex + ' should have aborted ' +
                      'but instead exited with exit code ' + exitCode +
                      ' and signal ' + signal;
       assert(common.nodeProcessAborted(exitCode, signal), errMsg);
     });
+
+    child.stdout.on('data', function(data) {
+      console.log(data);
+    });
   });
 }

[Trott]

BTW aren't there 10 functions in that array?

When I do console.log(tests.length), I get 13. And when I add logging in the child exit handler, I get 13 numbers (0 through 12). If you only get 10, that's curious....

[Trott]

Given how many processes this spawns, and given how we've seen timers be less reliable on heavily-loaded machines, and how some of these tests cases rely on timers, I wonder if the solution might be to move those tests to their own files. In a way, this is a test that is bypassing the way test.py regulates how many parallel jobs should run at once.

Other possible solution might be to rewrite this to run the tests one at a time and have each child process kick off the next one. But it will probably be easier code to understand to just have the tests in separate files.

@nodejs/testing

[joyeecheung]

There could be use cases like this test(spawning multiple processes to max CPU utilization, using --abort-on-uncaught-exception for coredumps in production), but given that domain is pending deprecation anyway, maybe it's OK to just let it go.

If there is no objection I'll be happy to pick up the splitting task.

When I do console.log(tests.length), I get 13. And when I add logging in the child exit handler, I get 13 numbers (0 through 12). If you only get 10, that's curious....

I get 10 when I console.log(tests.length) and can confirm this number counting them with my eyes...that's weird

[Trott]

@joyeecheung I figured it out. You are correctly looking at test-domain-no-error-handler-abort-on-uncaught.js and I am mistakenly looking at test-domain-abort-on-uncaught.js.

[Trott]

If there is no objection I'll be happy to pick up the splitting task.

@joyeecheung Yes, please feel free to split this test up into multiple test files to reduce concurrency!