AIX: parallel/test-stdio-closed

[mhdawson]

Version: master
Platform: AIX
Subsystem: io

parallel/test-stdio-closed fails on AIX, pulling this out of #7973 to avoid confusion since issues are not related.

not ok 855 parallel/test-stdio-closed # TODO : Fix flaky test
# 
# assert.js:85
#   throw new assert.AssertionError({
#   ^
# AssertionError: 1 == 42
#     at ChildProcess.<anonymous> (/home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/test/parallel/test-stdio-closed.js:25:10)
#     at ChildProcess.<anonymous> (/home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/test/common.js:416:15)
#     at emitTwo (events.js:106:13)
#     at ChildProcess.emit (events.js:191:7)
#     at Process.ChildProcess._handle.onexit (internal/child_process.js:215:12)
  ---
  duration_ms: 0.378
  ...

[Trott]

@nodejs/testing @Fishrock123

[Trott]

If it helps anyone troubleshoot or if it gives anyone an idea as to what's going on, the process.stdout.write() is failing (on AIX only) with:

EBADF: bad file descriptor, write

/cc gireeshpunathil who might have more information to add since their last comment.

[Trott]

Also, if it helps anyone, here's the modified version of the test I used to capture that info:

'use strict';
const common = require('../common');
const assert = require('assert');
const spawn = require('child_process').spawn;

if (common.isWindows) {
  common.skip('platform not supported.');
  return;
}

if (process.argv[2] === 'child') {
  try {
    process.stdout.write('stdout', function() {
      try {
        process.stderr.write('stderr', function() {
          process.exit(42);
        });
      } catch (e) {
        process.exit(200);
      }
    });
  } catch (e) {
    process.send(e.code);
    process.send(e.message);
    process.exit(20);
  }
  return;
}

// Run the script in a shell but close stdout and stderr.
const cmd = `"${process.execPath}" "${__filename}" child 1>&- 2>&-`;
const proc = spawn(
  '/bin/sh', ['-c', cmd], { stdio: ['ipc', 'inherit', 'inherit'] }
);

proc.on('message', (msg) => {
  console.log(msg);
});

proc.on('exit', common.mustCall(function(exitCode) {
  assert.strictEqual(exitCode, 42);
}));

Here's the results. I believe the EBADF is at the end because the test runner prints stderr first, then stdout. Probably should have used console.error() instead of console.log() but whatever.

# 
# assert.js:85
#   throw new assert.AssertionError({
#   ^
# AssertionError: 20 === 42
#     at ChildProcess.<anonymous> (/home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/test/parallel/test-stdio-closed.js:41:10)
#     at ChildProcess.<anonymous> (/home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/test/common.js:416:15)
#     at emitTwo (events.js:106:13)
#     at ChildProcess.emit (events.js:191:7)
#     at Process.ChildProcess._handle.onexit (internal/child_process.js:215:12)
# EBADF
# EBADF: bad file descriptor, write

[gireeshpunathil]

Thanks @Trott - this info will be really useful, I will investigate.

[gireeshpunathil]

ok, putting things into perspective:

Three processes: a node parent spawns a non-node child (sh) which closes its streams (1 and 2) and then spawns a node child. The grandchild attempts to write to its stream, and then returns an error code.

In Linux, the grandchild node apparently succeeded in writing into stdout, while AIX failed with EBADF (fd closed | invalid)

[gireeshpunathil]

So the question is, who closed the grandchild node's stream in AIX.

resorting to truss (AIX) and strace (Linux), I see that the parent shell does not perform fork and exec, instead only exec - means the closed streams of sh is inherited to the node grandchild:

Linux:

pid 18948] close(1)                    = 0
[pid 18948] execve("./node", ["./node", "k.js", "child"], [/* 30 vars */]) = 0

AIX:

3932586:        8913259: close(1)                               = 0
3932586:        8913259: execve("./node", 0x20010EF8, 0x20011428)  argc: 3

This is also visible if I add a timeout in the grandchild and examine the processes:

Linux:

bash-4.1$ ps
  PID TTY          TIME CMD
16440 pts/0    00:00:00 node
16446 pts/0    00:00:00 node
16457 pts/0    00:00:00 ps
17838 pts/0    00:00:00 csh
17855 pts/0    00:00:00 bash

AIX:

bash-4.3$ ps
     PID    TTY  TIME CMD
 5177436  pts/0  0:00 ps 
 5570730  pts/0  0:00 -ksh 
 3211596  pts/0  0:00 bash 
 2097736  pts/0  0:00 ./node k.js 
 2425576  pts/0  0:00 ./node k.js child 

As we can see, there is no /bin/sh in the list. That process itself exec'ed into node.

So, for the grandchild node, fd 1 is inherently closed.

[gireeshpunathil]

I wrote a C grandchild and used the same test case to invoke it to see how it behaves:

bash-4.3$ cat foo.c
#include <stdio.h>
#include  <stdlib.h>
int main() {
  fprintf(stdout, "hello\n");
  exit(42);
}

Linux:

fstat(1, 0x7fff2453d760)                = -1 EBADF (Bad file descriptor)
write(1, "hello\n", 6)                  = -1 EBADF (Bad file descriptor)
exit_group(42)                          = ?

AIX:

3735848:        11272557: kioctl(1, 22528, 0x00000000, 0x00000000)              = 0 Err#9  EBADF
3735848:        11272557: kwrite(1, " h e l l o\n", 6)          Err#9  EBADF
23735848:  11272557: _exit(42)

Here Linux and AIX behaved in the same manner, and the reason is well understood - the grandchild's fd is closed at inception.

So, Linux node seem to have special method to handle closed standard streams: either re-incarnate them through /dev/null device, or suppress the error code from propagation. AIX may be missing this.

At this point, I want to question the Linux behavior: Functionally, a node process spawned with standard streams suppressed, should behave in a console-less manner, and attempts to write to them should err. Any thoughts?

[Trott]

@gireeshpunathil node.cc has code in it designed to detect if any of the stdio streams are closed and (if so) open them on /dev/null.

This test is designed specifically to test that code. And it seems that either that code is not running on AIX (maybe the fstat() call works differently) or else the open() call in that code is returning the correct file descriptor but (for some reason) that file descriptor is not writable.

Any ideas?

[gireeshpunathil]

Thanks @Trott ! I was looking for such a location. I will now figure out what is happening out there in AIX.

[veshboo]

@gireeshpunathil What is the result without spawn on aix?

$ node -e "process.stdout.write('aaa')" 1>&-

@Trott (This is just guess) The code with fstat() is surrounded by #ifdef __POSIX__. Not defined in the aix build?

EDIT: I also found g++ ... '-D__POSIX__' ../src/node.cc line from CI console output. So forget about my guess.

[addaleax]

The code with fstat() is surrounded by #ifdef __POSIX__. Not defined in the aix build?

__POSIX__ should be defined for all non-Windows platforms, according to node.gyp.