Failures due to java.nio.channels.ClosedChannelException, AIX specific?

[sam-github]

I'm not sure if this is AIX specific, I think not, so please link non-AIX issues here if anyone finds them.

• jenkins build number: ppc machine node #

Examples from

• 23977: 3

21:03:15   ccache /home/iojs/gcc-6.3.0-1/opt/freeware/bin/g++ -o /home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/out/Release/obj.target/icuucx/deps/icu-small/source/common/ucnvsel.o ../deps/icu-small/source/common/ucnvsel.cpp '-DV8_DEPRECATION_WARNINGS' '-DV8_IMMINENT_DEPRECATION_WARNINGS' '-DOPENSSL_THREADS' '-DU_COMMON_IMPLEMENTATION=1' '-DU_ATTRIBUTE_DEPRECATED=' '-D_CRT_SECURE_NO_DEPRECATE=' '-DU_STATIC_IMPLEMENTATION=1' '-DUCONFIG_NO_SERVICE=1' '-DU_ENABLE_DYLOAD=0' '-DU_HAVE_STD_STRING=1' '-DUCONFIG_NO_BREAK_ITERATION=0' -I../deps/icu-small/source/common  -pthread -Wall -Wextra -Wno-unused-parameter -maix64 -Wno-deprecated-declarations -Wno-strict-aliasing -O3 -fno-omit-frame-pointer -fno-exceptions -std=gnu++1y -frtti -MMD -MF /home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/out/Release/.deps//home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/out/Release/obj.target/icuucx/deps/icu-small/source/common/ucnvsel.o.d.raw   -c
21:03:21 gmake[2]: *** Deleting file `/home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/out/Release/obj.target/icuucx/deps/icu-small/source/common/ucnvsel.o'
21:03:21 gmake[2]: *** Deleting file `/home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/out/Release/obj.target/icuucx/deps/icu-small/source/common/ucnvscsu.o'
21:03:21 gmake[2]: *** Deleting file `/home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/out/Release/obj.target/icuucx/deps/icu-small/source/common/ucnvmbcs.o'
21:03:21 gmake[2]: *** Deleting file `/home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/out/Release/obj.target/icuucx/deps/icu-small/source/common/ucnvlat1.o'
21:03:21 gmake[2]: *** Deleting file `/home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/out/Release/obj.target/icuucx/deps/icu-small/source/common/ucnvhz.o'
21:03:21 gmake[2]: *** [/home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/out/Release/obj.target/icuucx/deps/icu-small/source/common/ucnvsel.o] Terminated
21:03:21 gmake[2]: *** [/home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/out/Release/obj.target/icuucx/deps/icu-small/source/common/ucnvlat1.o] Terminated
21:03:21 gmake[2]: *** [/home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/out/Release/obj.target/icuucx/deps/icu-small/source/common/ucnvhz.o] Terminated
21:03:22 Build step 'Execute shell' marked build as failure
21:03:22 FATAL: java.nio.channels.ClosedChannelException
21:03:22 java.nio.channels.ClosedChannelException
21:03:22 Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from power8-aix-nodejs4.osuosl.org/140.211.9.77:41790
21:03:22 		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743)
21:03:22 		at hudson.remoting.Request.call(Request.java:202)

• 23976: 3

20:20:44 ok 2410 addons/stringbytes-external-exceed-max/test-stringbytes-external-at-max
20:20:44   ---
20:20:44   duration_ms: 2.958
20:20:44   ...
20:23:33 FATAL: command execution failed
20:23:33 java.nio.channels.ClosedChannelException
20:23:33 	at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:209)
20:23:33 	at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:222)

• 23974: 1 was 14:58:01 java.nio.channels.ClosedChannelException
• 23973: 3 was ditto
• 23972: 3 was ditto
• 23970: 3 was ditto

[richardlau]

There's #1837 (comment) which was mac OS but no idea if it's the same or not (I don't think we're running JDK 12 on AIX?).

[sam-github]

/home/iojs/jdk8/jre/bin/java --- could that be excessively old?

[mhdawson]

In many cases JDK8 is still the defacto level used so I don't think it being old would be my first guess.

[sam-github]

I used errpt -a on https://ci.nodejs.org/job/node-test-commit-aix/nodes=aix61-ppc64/24573/ from nodejs/node#28826 (comment), and it looks like there is indeed an OOM killer being triggered.

I'll look into whether this can be fixed at the system config level, or whether the test itself is too aggressively overcommitting memory allocation.

---------------------------------------------------------------------------
LABEL:          PGSP_KILL
IDENTIFIER:     C5C09FFA

Date/Time:       Tue Jul 23 15:52:06 CDT 2019
Sequence Number: 642991
Machine Id:      00FA74164C00
Node Id:         power8-nodejs4
Class:           S
Type:            PERM
WPAR:            Global
Resource Name:   SYSVMM          

Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED

Probable Causes
SYSTEM RUNNING OUT OF PAGING SPACE

Failure Causes
INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM
PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE

        Recommended Actions
        DEFINE ADDITIONAL PAGING SPACE
        REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)

Detail Data
PROGRAM
make
USER'S PROCESS ID:
              11534420
PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS
        1240
---------------------------------------------------------------------------
LABEL:          PGSP_KILL
IDENTIFIER:     C5C09FFA

Date/Time:       Tue Jul 23 15:52:06 CDT 2019
Sequence Number: 642990
Machine Id:      00FA74164C00
Node Id:         power8-nodejs4
Class:           S
Type:            PERM
WPAR:            Global
Resource Name:   SYSVMM          

Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED

Probable Causes
SYSTEM RUNNING OUT OF PAGING SPACE

Failure Causes
INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM
PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE

        Recommended Actions
        DEFINE ADDITIONAL PAGING SPACE
        REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)

Detail Data
PROGRAM
getty
USER'S PROCESS ID:
               8454182
PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS
         260
---------------------------------------------------------------------------
LABEL:          PGSP_KILL
IDENTIFIER:     C5C09FFA

Date/Time:       Tue Jul 23 15:52:06 CDT 2019
Sequence Number: 642989
Machine Id:      00FA74164C00
Node Id:         power8-nodejs4
Class:           S
Type:            PERM
WPAR:            Global
Resource Name:   SYSVMM          

Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED

Probable Causes
SYSTEM RUNNING OUT OF PAGING SPACE

Failure Causes
INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM
PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE

        Recommended Actions
        DEFINE ADDITIONAL PAGING SPACE
        REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)

Detail Data
PROGRAM
writesrv
USER'S PROCESS ID:
               7995564
PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS
          84
---------------------------------------------------------------------------
LABEL:          PGSP_KILL
IDENTIFIER:     C5C09FFA

Date/Time:       Tue Jul 23 15:52:06 CDT 2019
Sequence Number: 642988
Machine Id:      00FA74164C00
Node Id:         power8-nodejs4
Class:           S
Type:            PERM
WPAR:            Global
Resource Name:   SYSVMM          

Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED

Probable Causes
SYSTEM RUNNING OUT OF PAGING SPACE

Failure Causes
INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM
PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE

        Recommended Actions
        DEFINE ADDITIONAL PAGING SPACE
        REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)

Detail Data
PROGRAM
uprintfd
USER'S PROCESS ID:
               6684682
PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS
          12
---------------------------------------------------------------------------
LABEL:          PGSP_KILL
IDENTIFIER:     C5C09FFA

Date/Time:       Tue Jul 23 15:52:05 CDT 2019
Sequence Number: 642987
Machine Id:      00FA74164C00
Node Id:         power8-nodejs4
Class:           S
Type:            PERM
WPAR:            Global
Resource Name:   SYSVMM          

Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED

Probable Causes
SYSTEM RUNNING OUT OF PAGING SPACE

Failure Causes
INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM
PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE

        Recommended Actions
        DEFINE ADDITIONAL PAGING SPACE
        REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)

Detail Data
PROGRAM
ksh
USER'S PROCESS ID:
               8126476
PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS
          84
---------------------------------------------------------------------------
LABEL:          PGSP_KILL
IDENTIFIER:     C5C09FFA

Date/Time:       Tue Jul 23 15:52:02 CDT 2019
Sequence Number: 642986
Machine Id:      00FA74164C00
Node Id:         power8-nodejs4
Class:           S
Type:            PERM
WPAR:            Global
Resource Name:   SYSVMM          

Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED

Probable Causes
SYSTEM RUNNING OUT OF PAGING SPACE

Failure Causes
INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM
PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE

        Recommended Actions
        DEFINE ADDITIONAL PAGING SPACE
        REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)

Detail Data
PROGRAM
node
USER'S PROCESS ID:
               4456584
PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS
      316928

[Trott]

FWIW, a few years ago, those stringbytes-external tests were JavaScript tests instead of addon tests. If they can reasonably be JS tests and the addons thing was a test optimization or something, I wouldn't mind seeing them moved to pummel and run as part of node-daily-master only. (That's assuming they're part of the problem here. I might be making an incorrect assumption.)

[sam-github]

I think they are part of the problem, but moving them to pummel will just make node-daily-master fail more regularly, won't it? They are skipped on master, near the top of my TODO list is figuring out how to unskip them. I haven't read them, I'll look tomorrow to see what they are doing, and check out the history.

[Trott]

I think they are part of the problem, but moving them to pummel will just make node-daily-master fail more regularly, won't it? They are skipped on master, near the top of my TODO list is figuring out how to unskip them. I haven't read them, I'll look tomorrow to see what they are doing, and check out the history.

Well, it may be cheating, but at least currently, the pummel tests in node-daily-master are run on a Linux host and nothing else. So it won't damage reliability (because I don't think these tests typically fail on Linux), but of course, reduces our platform coverage....