[PRIORITY] Persistent failures on Windows

[rvagg]

See https://jenkins-iojs.nodesource.com/job/iojs+any-pr+multi/215/nodes=iojs-win2012r2/console for sample output, consistent with 2008 build and consistent across runs at the moment too.

10 failures in total. These have gone uncaught for a little while because of a combination of:

• Jenkins
• Jenkins configuration problems/mistakes (mainly on my part)
• The fact that test-ci still doesn't work on any platform so we have to resort to more hacky means of making enough tests run
• Errors in vcbuild.bat that have meant we haven't been running the full suite, fixed in build: improve vcbuild.bat #998

So we've been getting lots of blue when they really should have been red; so totally off everyone's radar.

I'm can't assess the severity of these failures at a glance, nor can I see a single common theme that would point to something to address. When I have time I'll go back and find a run that had these passing so we can start a manual bisect at least.

_test-child-process-stdio-big-write-end fixed in #1008_

=== release test-child-process-stdio-big-write-end ===
Path: parallel/test-child-process-stdio-big-write-end
events.js:141
      throw er; // Unhandled 'error' event
            ^
Error: write ENOTSUP
    at exports._errnoException (util.js:734:11)
    at Socket._writeGeneric (net.js:668:26)
    at Socket._writev (net.js:682:8)
    at doWrite (_stream_writable.js:293:12)
    at clearBuffer (_stream_writable.js:389:5)
    at onwrite (_stream_writable.js:338:7)
    at Socket.WritableState.onwrite (_stream_writable.js:88:5)
    at WriteWrap.afterWrite (net.js:765:12)
Command: c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\Release\iojs.exe c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\test\parallel\test-child-process-stdio-big-write-end.js

=== release test-timers-first-fire ===
Path: parallel/test-timers-first-fire
timer fired in -0.19568000000000296
assert.js:87
  throw new assert.AssertionError({
        ^
AssertionError: Timer fired early
    at null._onTimeout (c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\test\parallel\test-timers-first-fire.js:11:10)
    at Timer.listOnTimeout (timers.js:88:15)
Command: c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\Release\iojs.exe c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\test\parallel\test-timers-first-fire.js

=== release test-child-process-double-pipe ===
Path: parallel/test-child-process-double-pipe
grep stdin write 7
echo exit
grep stdin write 18
Command: c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\Release\iojs.exe c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\test\parallel\test-child-process-double-pipe.js
--- TIMEOUT ---

=== release test-child-process-exit-code ===
Path: parallel/test-child-process-exit-code
Command: c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\Release\iojs.exe c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\test\parallel\test-child-process-exit-code.js
--- TIMEOUT ---

=== release test-child-process-spawnsync ===
Path: parallel/test-child-process-spawnsync
sleep started
Command: c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\Release\iojs.exe c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\test\parallel\test-child-process-spawnsync.js
--- TIMEOUT ---

=== release test-child-process-spawnsync-input ===
Path: parallel/test-child-process-spawnsync-input
Command: c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\Release\iojs.exe c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\test\parallel\test-child-process-spawnsync-input.js
--- TIMEOUT ---

=== release test-http-curl-chunk-problem ===
Path: parallel/test-http-curl-chunk-problem
dd command:  "c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\Release\iojs.exe" "c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\test\fixtures\create-file.js" "c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\test\tmp.11\big" 10485760
Server running at http://localhost:8080
making curl request
Command: c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\Release\iojs.exe c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\test\parallel\test-http-curl-chunk-problem.js
--- TIMEOUT ---

=== release test-process-kill-null ===
Path: parallel/test-process-kill-null
Command: c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\Release\iojs.exe c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\test\parallel\test-process-kill-null.js
--- TIMEOUT ---

_test-pipe-head fixed in #1008_

=== release test-pipe-head ===
Path: sequential/test-pipe-head
Command: c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\Release\iojs.exe c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\test\sequential\test-pipe-head.js
--- TIMEOUT ---

=== release test-stdin-from-file ===
Path: sequential/test-stdin-from-file
"c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\Release\iojs.exe" "c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\test\fixtures\echo-close-check.js" < "c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\test\fixtures\stdin.txt"
c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\test\sequential\test-stdin-from-file.js:37
  if (err) throw err;
                 ^
Error: Command failed: C:\Windows\system32\cmd.exe /s /c ""c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\Release\iojs.exe" "c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\test\fixtures\echo-close-check.js" < "c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\test\fixtures\stdin.txt""


assert.js:87

  throw new assert.AssertionError({

        ^

AssertionError: false == true

    at Server.<anonymous> (c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\test\fixtures\echo-close-check.js:17:5)

    at Server.g (events.js:257:16)

    at emitNone (events.js:67:13)

    at Server.emit (events.js:163:7)

    at net.js:1184:12

    at process._tickCallback (node.js:350:11)


    at ChildProcess.exithandler (child_process.js:716:12)
    at emitTwo (events.js:87:13)
    at ChildProcess.emit (events.js:169:7)
    at maybeClose (child_process.js:984:16)
    at Process.ChildProcess._handle.onexit (child_process.js:1057:5)
Command: c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\Release\iojs.exe c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\test\sequential\test-stdin-from-file.js

[rvagg]

ping @iojs/platform-windows, this needs fairly urgent attention from anyone capable of digging in and figuring these out.

[rvagg]

Two of these have been fixed by #1008

Last complete test-simple run I can find is: https://jenkins-iojs.nodesource.com/job/iojs+any-pr+multi/nodes=iojs-win2012r2/54/console which is back at January 12th. i.e. so it seems that we've been happily seeing passes in CI for Windows builds since we've been "releasing" and none of us have noticed that it's been mainly test-message that's been running. This is entirely because it's been assumed that vcbuild.bat ... test-simple test-message is supposed to bundle both of them in to one, but unfortunately vcbuild.bat never had that properly working anyway. Fixed now in #998.

That means two things:

1. We're much further behind on Windows quality than we thought
2. These fixes probably aren't quite as urgent as we thought because some of these may be have been broken for longer than just 1.4.1 but we haven't had direct bug reports about them.

/cc @iojs/build

[rvagg]

I had limited success in reproducing many of these. Running them locally on a Windows machine and even running them on the exact same server that is giving us the failures via Jenkins, I can only make either 4 or 3 tests fail. The block of timeouts seem to be somehow Jenkins-specific.

Can other Windows users verify against 1.v HEAD and report back? vcbuild x64 release nosign test-simple (add test-message at the end to do exactly the same as Jenkins).

[domenic]

I get these failures: https://gist.github.com/domenic/ad191da152fc6632fa32 Unsure what to make of them compared to yours above.

I guess the first one, maybe others, is due to not having the openssl binary installed. I'll go check my prereqs and report back if anything else changes.

[rvagg]

Install git bash and you'll get what you need. Google "git windows" and download and install that one.

[domenic]

Yeah I have Git bash but when I try to run .bat files from within it all hell breaks loose.

[rvagg]

You should be able to run vcbuild.bat from cmd.exe but git bash comes with curl and other Unix utils that can go in your PATH. I think that's an install option though.

[domenic]

OK yeah, getting the same failures minus the first one (the openssl one).

[seishun]

I have 32-bit Windows at home so I ran vcbuild release nosign test-simple against b72fa03, The result is similar to @domenic's: https://gist.github.com/seishun/ebba8198423259444892

After deleting tmp.0 manually I re-ran parallel/test-fs-access and it passed.

[rvagg]

git bisect says that @indutny's changes in b968623 / #840 introduced the test-stdin-from-file failures

I'm still having trouble reproducing the timeout failures, perhaps something odd about the way the processes are being spawned via Jenkins is feeding in to that.