What is the cause of "Build timed out"?

[sam-github]

An aix61-ppc64 nightly failed, with no XML test results. Can someone point me to the timeout? I'm a bit confused, I know tools/test.py will timeout a test if it takes too long, but that cause a tap entry indicating the test that failed, what is this extra timeout that occurs and causes no test results to be reported? And what causes it, is it a bug in the test.py tool? Is it a too low jenkins timeout?

See https://ci.nodejs.org/job/node-test-commit-aix/23557/nodes=aix61-ppc64/:

00:12:31 ok 2535 message/vm_dont_display_syntax_error
00:12:31   ---
00:12:31   duration_ms: 0.108
00:12:31   ...
00:22:31 Build timed out (after 10 minutes). Marking the build as failed.
00:22:31 gmake[1]: *** [test-ci] Terminated
00:22:31 gmake[1]: Leaving directory `/home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64'
00:22:31 gmake: *** [run-ci] Error 2
00:22:31 Build was aborted

• https://ci.nodejs.org/job/node-test-commit-aix/23557/nodes=aix61-ppc64/console

[refack]

This is the symptom of #1820. The test harness stalls while trying to setup the tested process. The timeout is triggered by Jenkins which is configured to kill jobs that show no activity for a certain time. For AIX it's 600s - https://ci.nodejs.org/job/node-test-commit-aix/configure under "Build Environment".

[sam-github]

#1820 could be one cause of this, but its not unique to AIX, it just happened to me on Windows. Its entirely possible there are multiple things happening, depending on platform, ending in the same symptom.

11:15:06 ok 1265 sequential/test-vm-timeout-rethrow
11:15:06   ---
11:15:06   duration_ms: 0.248
11:15:06   ...
11:30:06 Build timed out (after 15 minutes). Marking the build as failed.
11:30:07 Build was aborted
11:30:07 Run condition [Always] enabling perform for step [[Execute Windows batch command, Execute shell]]

• https://ci.nodejs.org/job/node-test-binary-windows-2/1319/COMPILED_BY=vs2017,RUNNER=win2008r2-vs2017,RUN_SUBSET=1/console from https://ci.nodejs.org/job/node-test-pull-request/23699/ from Cleanup old ssl code node#28085

[refack]

it just happened to me on Windows. Its entirely possible there are multiple things happening, depending on platform, ending in the same symptom.

Indeed, I've seen that before, on Windows AFAICT it has a different cause. One of the tested procs spawns a child that does not exit, and thus blocking the python harness (I seem to have misplaced the cmd of that process that I wrote down, but AFAIR it was not obvious to which test it is related)

[sam-github]

@refack have you ever seen it locally on Windows, or just CI? I'm wondering where someone interested in fixing this on Windows (if I can find a volunteer) would start?

[refack]

Only on CI, and only on Windows2008R2 which BTW is going out of support at the end of the year (also only platform we see nodejs/node#20750). I think that's why those issues are hard to get priority for...

[sam-github]

Hard to fix, but pretty frequent, https://ci.nodejs.org/job/node-test-binary-windows-2/1327/COMPILED_BY=vs2017,RUNNER=win2008r2-vs2017,RUN_SUBSET=1/console failed on this, too.

[sam-github]

Hm, this happens on lots of platforms, here's a linux one, but this happened during the build phase:

19:35:23   ccache g++ -o /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-last-latest-x64/out/Release/obj.target/v8_base_without_compiler/deps/v8/src/builtins/builtins-weak-refs.o ../deps/v8/src/builtins/builtins-weak-refs.cc '-DV8_GYP_BUILD' '-DV8_TYPED_ARRAY_MAX_SIZE_IN_HEAP=64' '-DOPENSSL_THREADS' '-DV8_TARGET_ARCH_X64' '-DV8_EMBEDDER_STRING="-node.13"' '-DENABLE_DISASSEMBLER' '-DV8_PROMISE_INTERNAL_FIELD_COUNT=1' '-DENABLE_MINOR_MC' '-DV8_INTL_SUPPORT' '-DV8_USE_SNAPSHOT' '-DV8_CONCURRENT_MARKING' '-DV8_EMBEDDED_BUILTINS' '-DV8_USE_SIPHASH' '-DDISABLE_UNTRUSTED_CODE_MITIGATIONS' '-DV8_WIN64_UNWINDING_INFO' '-DICU_UTIL_DATA_IMPL=ICU_UTIL_DATA_STATIC' '-DUCONFIG_NO_SERVICE=1' '-DU_ENABLE_DYLOAD=0' '-DU_STATIC_IMPLEMENTATION=1' '-DU_HAVE_STD_STRING=1' '-DUCONFIG_NO_BREAK_ITERATION=0' -I../deps/v8 -I../deps/v8/include -I/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-last-latest-x64/out/Release/obj/gen/inspector-generated-output-root -I/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-last-latest-x64/out/Release/obj/gen/torque-output-root -I/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-last-latest-x64/out/Release/obj/gen/generate-bytecode-output-root -I../deps/icu-small/source/common -I../deps/icu-small/source/i18n -I../deps/icu-small/source/tools/toolutil  -pthread -Wno-unused-parameter -m64 -Wno-return-type -fno-strict-aliasing -m64 -O3 -fno-omit-frame-pointer -fdata-sections -ffunction-sections -O3 -fno-rtti -fno-exceptions -std=gnu++1y -MMD -MF /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-last-latest-x64/out/Release/.deps//home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-last-latest-x64/out/Release/obj.target/v8_base_without_compiler/deps/v8/src/builtins/builtins-weak-refs.o.d.raw   -c
21:35:23 Build timed out (after 120 minutes). Marking the build as failed.
21:35:23 Build was aborted

• https://ci.nodejs.org/job/node-test-commit-linux/nodes=fedora-last-latest-x64/28148/console

[joaocgreis]

On Windows (at least on recent runs) this only happened in https://ci.nodejs.org/computer/test-rackspace-win2008r2-x64-3/. For now I've took it offline, will try to understand what was happening there or recreate that machine.

This seems indeed to be a problem with node processes left running. parallel\test-child-process-exec-timeout leaves the child process running, parallel/test-child-process-execsync-maxbuf and parallel/test-child-process-execfile are somehow blocked by it (complete when it is killed - but have no node.exe running). Needs more investigation.

I've improved the diagnostics script in node-test-binary-windows-2 to show this. If this happens again somewhere else, please log into the machine and backup c:\jenkins_diagnostics.txt.

[joaocgreis]

This happens in other machines as well. Once nodejs/node#28723 lands I'll bring test-rackspace-win2008r2-x64-3 back online and see if it happens again on Windows.