Edge hangs when using karma-coverage reporter

[falsandtru]

Cannot finish the tests. In the following repo, gulp watch works well, but gulp test hang own process. When I remove Edge from target browsers, gulp test has no problem.

0. clone https://github.com/falsandtru/spica
1. npm i
2. gulp test

Note: karma start --browsers Edge --single-run works well.

[nickserv]

I originally didn't think this issue was specific to running it through Gulp, but after running .\node_modules\.bin\karma start --browsers Edge --single-run in your project it seems to work fine. This could be an issue with async code, or some other configuration in the gulpfile. It would help if we could isolate the problem more.

[nickserv]

I did a bunch of trial and error testing and I have narrowed this issue down a bit. It's not specific to Gulp, it happens when Edge is run together with the coverage reporter and processors. Whenever I run gulp test (which uses the coverage reporter), everything works fine until Edge gets to the point where one of the Edge tabs reports "localhost is not responding". This causes a stall in communication back to Karma which then causes Karma to kill the browser.

gulp test fails because thekarma:test task called by it uses coverage, but gulp watch doesn't since it doesn't enable coverage. gulp test should work for you if you remove coverage from the reporters list in the configuration for the karma:test task. Unfortunately the browserNoActivityTimeout and browserDisconnectTolerance options for Karma don't seem to help here, since the browser consistently stalls indefinitely (so relaunching it would just cause the same failure again).

This launcher plugin is somewhat unusual since it starts a helper exe (edge-launcher) instead of the browser itself (this is necessary to keep the process open since Edge itself isn't an exe), so this might be causing issues. I'm not sure what exactly is causing Edge to stall with this configuration, but it's probably an issue with karma-coverage, Edge itself, or the helper exe my plugin uses to launch Edge (edge-launcher with the -k flag for keep alive).

[falsandtru]

Thanks for the investigation! As a note, I reported the problem caused by coverage also with karma: karma-runner/karma#1768

[nickserv]

That's interesting, it seems like it was fixed a while ago in Karma 0.13.7. Is it possible that one of spica's dependencies is using an older version of Karma?

[falsandtru]

spica@0.0.1 uses karma@0.13.22. Another, gulp test is completed with https://github.com/falsandtru/pjax-api. So this bug is mutable.

[nickserv]

I'm not sure what you mean by that, are you saying this bug is unrelated?

[falsandtru]

spica and pjax-api uses the (mostly) same test config. But this bug is caused only on spica. So I think the investigation of the difference between these repos would be helpful for your debugging.

[nickserv]

I'm getting a timeout failure on Edge for pjax-api: Edge 14.14393.0 (Windows 10 0.0.0) Unit: layer/domain/router/module/fetch/xhr xhr timeout FAILED

[falsandtru]

Sorry, it is the unstable test. It is difficult to make a timeout of XHR. Please retry the test a few times.

[nickserv]

Ah you're right, the XHR failures seem to be somewhat random (I've seen similar issues before). They passed on my 4th run. I also noticed that this project is using the coverage reporter, and Edge doesn't hang when it fails the tests (unlike with spica). I'm starting to think this is a bug related to coverage generated code from spica specifically (likely a bug in karma-coverage or instanbul and not spica itself).

[nickserv]

I can't seem to produce this in a project other than spica, and spica is using karma-coverage 1.1.1 (which depends on karma 0.13.7, and should have fixed this issue). It seems like this would be worth a bug report to karma-coverage, do you agree?

[falsandtru]

At the head commit, spica uses karma@1.3.0, and karma-coverage support that version. Even so that, this bug seems karma-coverage's one?

[nickserv]

I think so, isn't it generating the JS that causes Edge to stall?

[falsandtru]

I tried to run gulp watch with karma-coverage using the same config 'dist/*.js': ['coverage', 'espower'] as gulp test, then Edge didn't stall. So the generated JS doesn't seem like the mainly cause.

[nickserv]

Whoops, I think I meant generating the reports (with the reporter option instead of the preprocessors option).

[falsandtru]

Indeed, then Edge stall the test. Because I cannot explain well why this is the karma-coverage's bug, can you open the issue in the repo of karma-coverage?

[falsandtru]

Note: When tests are run on debug page http://localhost:9876/debug.html, it is finished with no errors.

[nickserv]

That's interesting, what is different with the debug page? Does it disable coverage features?

[falsandtru]

It is enabled coverage features.

Here is the repro:

1. Use the following config.
2. Run gulp watch
3. Open the debug page.

gulp.task('karma:watch', function (done) {
  new Server({
    configFile: __dirname + '/karma.conf.js',
    browsers: config.browsers,
    reporters: ['dots', 'coverage'],
    preprocessors: {
      'dist/*.js': ['coverage', 'espower']
    },
  }, done).start();
});

[falsandtru]

I got the following error when I make the test failures with gulp test.

Edge 14.14393.0 (Windows 10 0.0.0) LOG: 'lib/fingerprint', 176025428
...........................................Missing error handler on `socket`.
TypeError: result.suite.join is not a function
    at DotsReporter.BaseReporter.specFailure (...\spica\node_modules\karma\lib\reporters\base.js:100:33)
    at DotsReporter.BaseReporter.onSpecComplete (...\spica\node_modules\karma\lib\reporters\base.js:85:12)
    at Server.<anonymous> (...\spica\node_modules\karma\lib\events.js:13:22)
    at emitTwo (events.js:111:20)
    at Server.emit (events.js:191:7)
    at Browser.onResult (...\spica\node_modules\karma\lib\browser.js:217:13)
    at Socket.<anonymous> (...\spica\node_modules\karma\lib\events.js:13:22)
    at emitOne (events.js:101:20)
    at Socket.emit (events.js:188:7)
    at Socket.onevent (...\spica\node_modules\socket.io\lib\socket.js:335:8)
    at Socket.onpacket (...\spica\node_modules\socket.io\lib\socket.js:295:12)
    at Client.ondecoded (...\spica\node_modules\socket.io\lib\client.js:193:14)
    at Decoder.Emitter.emit (...\spica\node_modules\component-emitter\index.js:134:20)
    at Decoder.add (...\spica\node_modules\socket.io-parser\index.js:247:12)
    at Client.ondata (...\spica\node_modules\socket.io\lib\client.js:175:18)
    at emitOne (events.js:96:13)
    at Socket.emit (events.js:188:7)
06 11 2016 16:14:38.126:WARN [Edge 14.14393.0 (Windows 10 0.0.0)]: Disconnected (1 times)

[nickserv]

Closing because I am deprecating this repo in favor of karma-runner/karma-edge-launcher (which is the same project, just moved to their org).

If you still have this issue please either create an issue there or ask me if you'd like me to create an issue linking to this one for you.