No coverage report generated for large project

[cklogs]

Hi there,

This library is now recommended for code coverage when using v11 of the angular-cli. We are currently using karma-coverage-istanbul-reporter because we have yet to get successful coverage reports with karma-coverage. Seeing, however, that there are deprecation warnings when using the istanbul reporter library with the v11 angular-cli, we would very much like to move off it.

Perhaps someone here can help discern what is wrong with our configuration or point us in the right direction.

I will include the karma.config, and test output debug logs for two different projects.

The first project (PROJECT-1) is a bare project that was initialized with ng new command. It was updated to Angular 11 using the ng update command from angular-cli. The point of including this project is to serve as a 'control' for the second project.

The second project (PROJECT-2) comes from a relatively large production codebase. In this codebase is roughly 5000 unit tests.

Important points

1. The angular.json test configuration block, tsconfig.spec.ts, and karma.config for both PROJECT-1 and PROJECT-2 are exactly the same. I will post the shared karma.config below.
2. v11 angular-cli still has memory management issues at build time, similar to those mentioned here. For build/run/test on PROJECT-2 we have to use the --max_old_space_size flag. Note that we are running with latest version of the angular-cli and node v12.x.

PROJECT-1 tests are run with: ng test.
PROJECT-2 tests are run with: node --max_old_space_size=8192 node_modules/@angular/cli/bin/ng test.

3. When running tests on PROJECT-1, I get consistent coverage reports. I never see coverage reports for PROJECT-2.
4. The only discernable difference between PROJECT-1 and PROJECT-2 is the --max_old_space_size configuration property mentioned about.

Below is the karma config file shared by the two projects and the debug logs that are output when running tests.

Karma config

// Karma configuration file, see link for more information
// https://karma-runner.github.io/1.0/config/configuration-file.html

module.exports = function (config) {
  config.set({
    basePath: '',
    frameworks: ['jasmine', '@angular-devkit/build-angular'],
    plugins: [
      require('karma-spec-reporter'),
      require('karma-jasmine'),
      require('karma-chrome-launcher'),
      require('karma-coverage'),
      require('@angular-devkit/build-angular/plugins/karma')
    ],
    client: {
      clearContext: false // leave Jasmine Spec Runner output visible in browser
    },
    coverage: {
      dir: require('path').join(__dirname, '../coverage/karma-coverage'),
      reports: ['html', 'lcovonly', 'text-summary']
    },
    reporters: ['dots', 'spec', 'coverage'],
    specReporter: {
	maxLogLines: 5, // limit number of lines logged per test
	suppressErrorSummary: true, // do not print error summary
	suppressFailed: false, // do not print information about failed tests
	suppressPassed: true, // do not print information about passed tests
	suppressSkipped: true, // do not print information about skipped tests
	showSpecTiming: false, // print the time elapsed for each spec
	failFast: false // test would finish with error when a first fail occurs.
     },
    customLaunchers: {
	'CustomChromeHeadless': {
		base: 'ChromeHeadless',
		flags: [
			'--disable-gpu',
			'--disable-web-security',
			'--no-sandbox',
			'--remote-debugging-port=9222'
		],
		debug: true
	}
    },
	browsers: ['CustomChromeHeadless'],
	browserDisconnectTolerance: 2,
	browserNoActivityTimeout: 300000, // five minutes
	browserDisconnectTimeout: 60000,
	captureTimeout: 300000,
	hostname: '127.0.0.1',
	port: 9876,
	colors: true,
	logLevel: config.LOG_DEBUG,
	autoWatch: false,
	singleRun: true,
	restartOnFileChange: false
  });
};

PROJECT-1-debug-output.txt
PROJECT-2-debug-output.txt

In the PROJECT-2 output you will see only ~20 tests ran. This is because I used an fdescribe to run a single test file so as to reduce the noise a little. The same result--no coverage--happens with or without this change.

The point of failure seems to be here:

Thanks in advance for your support.

[alan-agius4]

@cklogs, karma-coverage needs to be configured using the coverageReporter property, which is missing from your configuration.

Can you please try to change the configuration like the below;

// Karma configuration file, see link for more information
// https://karma-runner.github.io/1.0/config/configuration-file.html

module.exports = function (config) {
    config.set({
        basePath: '',
        frameworks: ['jasmine', '@angular-devkit/build-angular'],
        plugins: [
            require('karma-spec-reporter'),
            require('karma-jasmine'),
            require('karma-chrome-launcher'),
            require('karma-coverage'),
            require('@angular-devkit/build-angular/plugins/karma')
        ],
        client: {
            clearContext: false // leave Jasmine Spec Runner output visible in browser
        },
+       coverageReporter: {
+           dir: require('path').join(__dirname, './coverage/karma-coverage'),
+           subdir: '.',
+           reporters: [
+               { type: 'html' },
+               { type: 'text-summary' }
+            ]
+        },
-       coverage: {
-           dir: require('path').join(__dirname, '../coverage/karma-coverage'),
-           reports: ['html', 'lcovonly', 'text-summary']
-       },
        reporters: ['dots', 'spec', 'coverage'],
        specReporter: {
            maxLogLines: 5, // limit number of lines logged per test
            suppressErrorSummary: true, // do not print error summary
            suppressFailed: false, // do not print information about failed tests
            suppressPassed: true, // do not print information about passed tests
            suppressSkipped: true, // do not print information about skipped tests
            showSpecTiming: false, // print the time elapsed for each spec
            failFast: false // test would finish with error when a first fail occurs.
        },
        customLaunchers: {
            'CustomChromeHeadless': {
                base: 'ChromeHeadless',
                flags: [
                    '--disable-gpu',
                    '--disable-web-security',
                    '--no-sandbox',
                    '--remote-debugging-port=9222'
                ],
                debug: true
            }
        },
        browsers: ['CustomChromeHeadless'],
        browserDisconnectTolerance: 2,
        browserNoActivityTimeout: 300000, // five minutes
        browserDisconnectTimeout: 60000,
        captureTimeout: 300000,
        hostname: '127.0.0.1',
        port: 9876,
        colors: true,
        logLevel: config.LOG_DEBUG,
        autoWatch: false,
        singleRun: true,
        restartOnFileChange: false
    });
};

More information about karma-coverage configuration can be found here: https://github.com/karma-runner/karma-coverage/blob/master/docs/configuration.md

[cklogs]

@alan-agius4

I tried exactly as is suggested above and coverage reports are still not being generated.
Perhaps there is something more I'm missing?

[alan-agius4]

Without a reproduction it's hard to tell what's happening. You can try to debug

karma-coverage/lib/reporter.js

Line 224 in 1d34a56

this.executeReport = async function (reporterConfig, browser) {

to see what's happening.

[cklogs]

I'm still trying to demystify what is going on but I have observed the following:
For these particular methods:

karma-coverage/lib/reporter.js

Lines 185 to 201 in 1d34a56

	this.onRunStart = function (browsers) {
	coverageMaps = Object.create(null)
	// TODO(vojta): remove once we don't care about Karma 0.10
	if (browsers) {
	browsers.forEach(this.onBrowserStart.bind(this))
	}
	}
	this.onBrowserStart = function (browser) {
	var startingMap = {}
	if (includeAllSources) {
	startingMap = globalCoverageMap.get()
	}
	coverageMaps[browser.id] = istanbulLibCoverage.createCoverageMap(startingMap)
	}

The browsers object on line 185 is defined as:

BrowserCollection {
  browsers: [],
  emitter: EventEmitter {
    _events: [Object: null prototype] {},
    _eventsCount: 0,
    _maxListeners: undefined,
    [Symbol(kCapture)]: false
  }
}

This means that if (browsers) { executes.

However, this.onBrowserStart is never called during the entire execution of the tests.

Once the tests have all ran, this function is called:

karma-coverage/lib/reporter.js

Lines 203 to 210 in 1d34a56

	this.onBrowserComplete = function (browser, result) {
	var coverageMap = coverageMaps[browser.id]
	if (!coverageMap) return
	if (!result || !result.coverage) return
	coverageMap.merge(result.coverage)
	}

The parameter browser is defined as:

Browser {
  id: '35804940',
  fullName: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/89.0.4389.90 Safari/537.36',
  name: 'Chrome Headless 89.0.4389.90 (Windows 10)',
  lastResult: BrowserResult {
    startTime: 1617310253880,
    total: 0,
    success: 6032,
    failed: 0,
    skipped: 0,
    totalTime: 628830,
    netTime: 437749,
    error: false,
    disconnected: false
  },
  disconnectsCount: 0,
  activeSockets: [
    Socket {
      _events: [Object: null prototype],
      _eventsCount: 8,
      _maxListeners: undefined,
      nsp: [Namespace],
      client: [Client],
      acks: Map {},
      fns: [],
      flags: {},
      _rooms: Set {},
      server: [Server],
      adapter: [Adapter],
      id: 'eEVk_hCsxP6b3WzbAAAD',
      connected: true,
      disconnected: false,
      handshake: [Object],
      [Symbol(kCapture)]: false
    }
  ],
  noActivityTimeout: 300000,
  singleRun: true,
  clientConfig: {
    args: [],
    useIframe: true,
    runInParent: false,
    captureConsole: true,
    clearContext: true,
    originalArgs: []
  },
  collection: BrowserCollection {
    browsers: [ [Circular] ],
    emitter: Server {
      _events: [Object: null prototype],
      _eventsCount: 17,
      _maxListeners: undefined,
      log: [Logger],
      loadErrors: [],
      _injector: [Injector],
      _boundServer: [Server],
      _fileList: [FileList],
      [Symbol(kCapture)]: false
    }
  },
  emitter: Server {
    _events: [Object: null prototype] {
      load_error: [Function],
      exit: [Array],
      run_complete: [Array],
      browser_complete: [Array],
      file_list_modified: [Array],
      run_start: [Array],
      browser_start: [Array],
      browser_error: [Array],
      browser_log: [Array],
      spec_complete: [Array],
      spec_failure: [Function],
      browsers_change: [Function],
      browser_register: [Function],
      stop: [Function],
      browser_restart_failure: [Function],
      browser_complete_with_no_more_retries: [Function],
      browser_process_failure: [Function]
    },
    _eventsCount: 17,
    _maxListeners: undefined,
    log: Logger {
      category: 'karma-server',
      context: {},
      parseCallStack: [Function: defaultParseCallStack]
    },
    loadErrors: [],
    _injector: Injector {
      _providers: [Object: null prototype],
      _instances: [Object: null prototype],
      get: [Function: get],
      invoke: [Function: invoke],
      instantiate: [Function: instantiate],
      createChild: [Function: createChild]
    },
    _boundServer: Server {
      _events: [Object: null prototype],
      _eventsCount: 3,
      _maxListeners: undefined,
      _connections: 0,
      _handle: [TCP],
      _usingWorkers: false,
      _workers: [],
      _unref: false,
      allowHalfOpen: false,
      pauseOnConnect: false,
      _connectionKey: '4:0.0.0.0:9876',
      [Symbol(kCapture)]: false,
      [Symbol(asyncId)]: 13
    },
    _fileList: FileList {
      _patterns: [Array],
      _excludes: [Array],
      _emitter: [Circular],
      _preprocess: [AsyncFunction: preprocess],
      buckets: [Map],
      _refreshing: [Promise],
      _emitModified: [Function]
    },
    [Symbol(kCapture)]: false
  },
  socket: Socket {
    _events: [Object: null prototype] {
      start: [Function],
      info: [Function],
      karma_error: [Function],
      result: [Function],
      complete: [Function],
      error: [Function],
      register: [Function],
      disconnect: [Function]
    },
    _eventsCount: 8,
    _maxListeners: undefined,
    nsp: Namespace {
      _events: [Object: null prototype],
      _eventsCount: 1,
      _maxListeners: undefined,
      sockets: [Map],
      _fns: [],
      _rooms: Set {},
      _flags: {},
      _ids: 0,
      server: [Server],
      name: '/',
      adapter: [Adapter],
      [Symbol(kCapture)]: false
    },
    client: Client {
      sockets: Map {},
      nsps: Map {},
      server: [Server],
      conn: [Socket],
      encoder: Encoder {},
      decoder: [Decoder],
      id: '2DlQ3r3gH89vkFHCAAAA',
      onclose: [Function: bound onclose],
      ondata: [Function: bound ondata],
      onerror: [Function: bound onerror],
      ondecoded: [Function: bound ondecoded],
      connectTimeout: undefined
    },
    acks: Map {},
    fns: [],
    flags: {},
    _rooms: Set {},
    server: Server {
      _events: [Object: null prototype] {},
      _eventsCount: 0,
      _maxListeners: undefined,
      _nsps: [Map],
      parentNsps: Map {},
      _path: '/socket.io',
      clientPathRegex: /^\/socket\.io\/socket\.io(\.min|\.msgpack\.min)?\.js(\.map)?$/,
      _connectTimeout: 45000,
      _serveClient: true,
      _parser: [Object],
      encoder: Encoder {},
      _adapter: [class Adapter extends EventEmitter],
      sockets: [Namespace],
      opts: [Object],
      eio: [Server],
      httpServer: [Server],
      engine: [Server],
      [Symbol(kCapture)]: false
    },
    adapter: Adapter {
      _events: [Object: null prototype] {},
      _eventsCount: 0,
      _maxListeners: undefined,
      nsp: [Namespace],
      rooms: [Map],
      sids: [Map],
      encoder: Encoder {},
      [Symbol(kCapture)]: false
    },
    id: 'tpLs3SFHdGeiHnz_AAAB',
    connected: false,
    disconnected: true,
    handshake: {
      headers: [Object],
      time: 'Thu Apr 01 2021 13:50:53 GMT-0700 (Pacific Daylight Time)',
      address: '127.0.0.1',
      xdomain: false,
      secure: false,
      issued: 1617310253837,
      url: '/socket.io/?EIO=4&transport=polling&t=NYFD0BQ',
      query: [Object: null prototype],
      auth: {}
    },
    [Symbol(kCapture)]: false
  },
  timer: {
    setTimeout: [Function: setTimeout],
    clearTimeout: [Function: clearTimeout]
  },
  disconnectDelay: 60000,
  log: Logger {
    category: 'Chrome Headless 89.0.4389.90 (Windows 10)',
    context: {},
    parseCallStack: [Function: defaultParseCallStack]
  },
  noActivityTimeoutId: Timeout {
    _idleTimeout: 300000,
    _idlePrev: [TimersList],
    _idleNext: [TimersList],
    _idleStart: 983207,
    _onTimeout: [Function],
    _timerArgs: undefined,
    _repeat: null,
    _destroyed: false,
    [Symbol(refed)]: true,
    [Symbol(asyncId)]: 1889860,
    [Symbol(triggerId)]: 1889856
  },
  pendingDisconnect: Timeout {
    _idleTimeout: -1,
    _idlePrev: null,
    _idleNext: null,
    _idleStart: 385875,
    _onTimeout: null,
    _timerArgs: undefined,
    _repeat: null,
    _destroyed: true,
    [Symbol(refed)]: true,
    [Symbol(asyncId)]: 1809518,
    [Symbol(triggerId)]: 1809514
  },
  state: 'CONNECTED'
}

coverageMaps are defined as:

[Object: null prototype] {}

The result is a large object containing coverage results for each file. One such example:

   'C:\\app\\shared\\helpers\\url-serializer.ts': {
      path: 'C:\\app\\shared\\helpers\\url-serializer.ts',
      statementMap: [Object],
      fnMap: [Object],
      branchMap: [Object],
      s: [Object],
      f: [Object],
      b: [Object],
      inputSourceMap: [Object],
      _coverageSchema: '1a1c01bbd47fc00a2c39e90264f33305004495a9',
      hash: 'b12799a1103f348e9c0f91bb2cb5bbf81c79d909'
    }

Is there anything more you would like to see?

[cklogs]

@alan-agius4 When you have a chance, could you please review my updates?
Advance thanks!

[alan-agius4]

I'll leave the triaging to someone from the karma-coverage team.

[cklogs]

To the maintainers of this repo:

I have a small suspicion that,

1. the sheer size of our project and,
2. the associated long compile time while running tests

have something to do with coverage failing to be generated. But this is only a hunch.

I cannot share or anonymize our code base, but if there is a large open repository that you would suggest I try and reproduce this issue from, I will happily try.

Aside from that, if you have any insights on the above issue, please let me know here or contact me directly.

Thanks again.

[WebMex]

We also have a big project with 3000+ unit tests. And I can see coverage reports generated only on powerful machines. On our ci server and on slow machines it's not generated.

[cklogs]

@WebMex
Thank you for the additional data point!

Do you plan to continue using karma-coverage, or have you sought out a different library? We have not been able to move onto karma-coverage (from karma-coverage-instanbul-reporter) because of the above issue, but we are open to other coverage generators.

[WebMex]

@cklogs I have rolled back to karma-coverage-instanbul-reporter so far and waiting for the fix. Despite the warning message from angular it is still possible to use it.

[cklogs]

@alan-agius4

So far I have had little luck getting feedback from any of the maintainers of this library.
Given karma-coverage is now the defacto standard coverage library for Angular, it may be necessary to raise this up to the greater Angular team.

Please advise on how to most effectively do this.

Thanks in advance for your help!

[cklogs]

@WebMex

Thanks to your tip, I tested karma-coverage on a faster box (Ryzen 7 PRO / 32 GB RAM), and I am getting consistent coverage results there. Still need this fix for our deployment servers to generate coverage, but I'm hoping these multiple data points give the maintainers a lead on the issue.

[alan-agius4]

This would be a hard one to track down without a reproduction. I tried it on a seed project and generated a large number of tests but I didn't manage to replicate.

Unfortunately, I am not too familiar with this plugin to get to the bottom of this without being able to reproduce the issue.

[arobinson]

I am seeing the same issue. If I run the test on my mac locally, I'm seeing the lcov.info file always get generated. However, running the test with even a greater --max_old_space_size size specified and run by Jenkins on a linux box intermittently does not generate the lcov.info file. There are no reports of exceptions or anything else I can see that explain why the report is not generated.

When the coverage isn't generated I see something like:

[2021-06-29T21:37:31.258Z] TOTAL: 6266 SUCCESS
[2021-06-29T21:37:31.258Z] 
[2021-06-29T21:37:31.258Z] Finished in 13 mins 7.178 secs / 9 mins 58.633 secs @ 21:37:31 GMT+0000 (Coordinated Universal Time)
[2021-06-29T21:37:31.258Z] 
[2021-06-29T21:37:31.258Z] SUMMARY:
[2021-06-29T21:37:31.258Z] ✔ 6266 tests completed
[2021-06-29T21:37:31.258Z] ℹ 28 tests skipped

When I run it locally and the file is generated I get:

TOTAL: 6266 SUCCESS

Finished in 14 mins 0.813 secs / 10 mins 58.549 secs @ 15:34:47 GMT-0600 (Mountain Daylight Time)

SUMMARY:
✔ 6266 tests completed
ℹ 28 tests skipped
29 06 2021 15:35:04.211:WARN [launcher]: Chrome was not killed in 2000 ms, sending SIGKILL.

Oddly, it seems the Chrome was not killed is usually reported when the lcov.info file is generated, but if that is not reported, the coverage file was not generated. So it almost seems like the node process is exiting before the karma-coverage plugin is done generating the report.

Could the problem be that the process can exit before the plugin/reporter has completed writing the file?

[cklogs]

@alan-agius4
Perhaps this project might be a good place to run a smoke test of this issue: https://github.com/angular/angular-cli-stress-test. I was able to get it roughly running just by dropping the files into a newly spun up Angular 12 project.
I could not get coverage reports generating on the codebase. You will likely be able to reproduce the issue so long as your machine is not too powerful.

[nukithelegend]

Please can this be done? We have the same issue. We have about 13000 unit tests. Sometimes the coverage report gets generated, however it mostly fails. We run them on a Linux docker image trion/ng-cli-karma.

It's upsetting to be "punished" for having too many tests.

[nukithelegend]

I have done a previous investigation on this, and I found that it was related to the speed/number of tests. It failed in Istanbul, I believe it was when merging the coverage summaries that the result was just suddenly empty and then no coverage were created

[arobinson]

Unfortunately this started happening ~75% of the time for us, but I could not reproduce it locally. Only solution I could find was to uninstall and go back to an unmaintained, but much more reliable solution for now. This is a blocker, we need it to be 100% reliable.

[arobinson]

This looks to be an issue with Karma:
karma-runner/karma#945

Unfortunately it seems that Karma is a dead project. That issue was opened in 2014 and was closed since no one worked on the issue

[jginsburgn]

It is absolutely not dead! We work with limited resources, so we cannot address every issue immediately and sadly that one got closed a long time ago; I will be adding it to my working queue. Also, your collaboration is welcome.

[lzilioli-prism]

I am experiencing the same issue after addressing the following message from our test run:

'karma-coverage-istanbul-reporter' usage has been deprecated since version 11.
            Please install 'karma-coverage' and update 'karma.conf.js.' For more info, see https://github.com/karma-runner/karma-coverage/blob/master/README.md

I am going to revert the change in my project, and follow this issue, in hopes it gets resolved soon.

Some additional observations: I have seen the coverage report generated one time successfully, and if I scope the files in my test runner to a smaller subset of .spec.ts files, it is generated every time. This clearly seems to be a memory or cpu issue. Whats troubling to me is that the process exits as though it successfully generated the report!

[lzilioli-prism]

Update: Reverting back to karma-coverage-istanbul-reporter 100% fixed the issue, and our coverage reports are being reliably generated again, despite the deprecation warning. I can confirm there are issues with code coverage reports and:

Angular:

$ ng --version

     _                      _                 ____ _     ___
    / \   _ __   __ _ _   _| | __ _ _ __     / ___| |   |_ _|
   / △ \ | '_ \ / _` | | | | |/ _` | '__|   | |   | |    | |
  / ___ \| | | | (_| | |_| | | (_| | |      | |___| |___ | |
 /_/   \_\_| |_|\__, |\__,_|_|\__,_|_|       \____|_____|___|
                |___/


Angular CLI: 12.2.1
Node: 14.15.5
Package Manager: npm 6.14.11
OS: darwin x64

Angular: 12.2.1
... animations, cdk, cli, common, compiler, compiler-cli, core
... forms, language-service, material, material-moment-adapter
... platform-browser, platform-browser-dynamic, platform-server
... router

Package                         Version
---------------------------------------------------------
@angular-devkit/architect       0.1202.1
@angular-devkit/build-angular   12.2.1
@angular-devkit/core            12.2.1
@angular-devkit/schematics      12.2.1
@angular/flex-layout            12.0.0-beta.34
@schematics/angular             12.2.1
rxjs                            6.6.7
typescript                      4.3.5
webpack                         5.50.0

karma stuffs from package.json:

$ cat package.json | grep karma
		"karma-spec-reporter": "^0.0.32",
		"karma": "~6.3.4",
		"karma-chai": "^0.1.0",
		"karma-chrome-launcher": "~3.1.0",
		"karma-cli": "^2.0.0",
		"karma-coverage": "~2.0.3",
		"karma-jasmine": "~4.0.0",
		"karma-jasmine-html-reporter": "^1.5.0",
		"karma-junit-reporter": "^2.0.1",
		"karma-phantomjs-launcher": "^1.0.4",
		"karma-typescript": "^5.5.1",

I hope this helps

[rrauenza]

I think I am seeing this problem as well. Intermittently, the coverage directory isn't created. I don't know that my project is large or not... not sure of the baseline!

$ ./node_modules/.bin/cloc --include-lang=TypeScript src/app/ vendored/
     381 text files.
     380 unique files.                                          
     148 files ignored.

github.com/AlDanial/cloc v 1.90  T=1.48 s (165.3 files/s, 117159.1 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
TypeScript                     245           2320           1451         169847
-------------------------------------------------------------------------------
SUM:                           245           2320           1451         169847
-------------------------------------------------------------------------------

An example of failure:

    ✓ should submit form
    ✓ should submit form API data
    ✓ should check permission for edit
    ✓ should edit & delete API key
TOTAL: 182 SUCCESS
TOTAL: 182 SUCCESS
Chrome Headless 94.0.4606.61 (Linux x86_64): Executed 182 of 0 (skipped 1) SUCCESS (59.589 secs / 17.681 secs)
TOTAL: 182 SUCCESS
 ____  _   _  ____ ____ _____ ____ ____  
/ ___|| | | |/ ___/ ___| ____/ ___/ ___| 
\___ \| | | | |  | |   |  _| \___ \___ \ 
 ___) | |_| | |__| |___| |___ ___) |__) |
|____/ \___/ \____\____|_____|____/____/ 
                                         
$ ls -la coverage/
ls: cannot access coverage/: No such file or directory
Cleaning up file based variables

An example of success:

Chrome Headless 94.0.4606.61 (Linux x86_64): Executed 182 of 183 (skipped 1) SUCCESS (17.465 secs / 15.649 secs)
TOTAL: 182 SUCCESS
TOTAL: 182 SUCCESS
Chrome Headless 94.0.4606.61 (Linux x86_64): Executed 182 of 183 (skipped 1) SUCCESS (17.465 secs / 15.649 secs)
TOTAL: 182 SUCCESS
08 10 2021 11:15:39.085:WARN [launcher]: ChromeHeadless was not killed in 2000 ms, sending SIGKILL.
 ____  _   _  ____ ____ _____ ____ ____  
/ ___|| | | |/ ___/ ___| ____/ ___/ ___| 
\___ \| | | | |  | |   |  _| \___ \___ \ 
 ___) | |_| | |__| |___| |___ ___) |__) |
|____/ \___/ \____\____|_____|____/____/ 
                                         
$ ls -la coverage/
total 832
drwxrwxr-x 4 gitlab-runner gitlab-runner    152 Oct  8 11:15 .
drwxrwxr-x 8 gitlab-runner gitlab-runner   4096 Oct  8 11:15 ..
-rw-rw-r-- 1 gitlab-runner gitlab-runner 658117 Oct  8 11:15 cobertura.txt
drwxrwxr-x 3 gitlab-runner gitlab-runner    182 Oct  8 11:15 report-html
drwxrwxr-x 3 gitlab-runner gitlab-runner     42 Oct  8 11:15 report-lcov
-rw-rw-r-- 1 gitlab-runner gitlab-runner 160020 Oct  8 11:15 report-lcovonly.txt
-rw-rw-r-- 1 gitlab-runner gitlab-runner    697 Oct  8 11:15 teamcity.txt
-rw-rw-r-- 1 gitlab-runner gitlab-runner    302 Oct  8 11:15 text-summary.txt
-rw-rw-r-- 1 gitlab-runner gitlab-runner  14499 Oct  8 11:15 text.txt

What is kind of interesting is that I get the SIGKILL in the case in which it works.... but maybe that's just karma killing the browser as it writes out the reports.

I also had no problems when using Istanbul previously.

Due to the warning, I'm just going to arbitrarily try setting the timeout higher in karma.conf.js and see what happens... I can't justify why it might help, but I'd like to see the warning go away anyway:

processKillTimeout: 60 * 1000

update: it didn't help. I still get intermittent failures to create the coverage output.
update 2: reducing the number of kinds of reports to just text/text-summary might have helped. Waiting for more data.
update 3: seems to still happen. Maybe when cicd machine is under higher load. Will need to go back to istanbul version more than likely.

[ozkoidi]

I have exactly the same issue. The coverage reports are generated only when the following messasge is logged:
ChromeHeadless was not killed in 2000 ms, sending SIGKILL.

Is there a way to force karma to always send this? Or any other updates on this issue?

[devoto13]

From a quick look it seems that the problem is that onRunComplete returns a Promise, but Karma does not wait for for it. As a result Karma shutdown will race with the writing of the coverage report.

Unfortunately, without a reproduction it is hard to confirm or reject the theory. I wonder if somebody having the problem can run tests using this branch and see if it solves the problem? The only change there is that Karma will not call process.exit once done, thus allowing all async activities to complete.

npm i devoto13/karma#test-coverage

UPD Looks like that promise is actually awaited in the onExit callback. Another place, which is not synchronized is this one 🤔 I think it still will be useful to perform the test, so we can confirm that the issue is caused by the race and focus on locating and fixing this race.

[arobinson]

I wonder if somebody having the problem can run tests using this branch and see if it solves the problem?

Trying now. I'll do a few runs on this and report back to see how it does with our code. Thanks

[ozkoidi]

@devoto13, thanks for your suggestion. I removed the karma-parallel package and the result is still the same, so no need for you to look into that package.

In case it helps the specs of the Microsoft hosted agents running the tests are: 2 core CPU, 7 GB of RAM, and 14 GB of SSD disk space. I think those VM should be enough to run the test properly but I don't know what else could make the test not generate coverage reports randomly. Updated logs:

With test coverage generated 👍

Without test coverage 👎

[devoto13]

@ozkoidi Thank you for the quick feedback. Can you please install karma and karma-coverage from these two branches and make more sample runs on your CI?

$ npm i devoto13/karma#test-coverage
$ npm i devoto13/karma-coverate#test-coverage

The first one removes the call to the process.exit on Karma shut down, so if the problem is resolved by the change, it means that we still have a race somewhere in the coverage generation routine. Per #434 (comment). Here is the code.

The second one is based on #434 (comment). It adds logs to all karma-coverage hooks so that we can compare good and bad runs and see if we can get some clue about what is different. Here is the code.

[ozkoidi]

Thank you @devoto13, I tried your branches yesterday and the result was the same (no coverage generated half the time).
With Karma's logLevel set to debug the logs are huge so I put here just the beginning and end of the two types of run.

I include the reduced and cleaned logs below but as far as I can see the differences are:

Coverage report generated	No coverage report generated
Logs show [coverage]: onBrowserComplete completed	No logs about onBrowserComplete being called
In "[coverage]: onRunComplete called BrowserCollection" the Browser object's pendingDisconnect property is null	In "[coverage]: onRunComplete called BrowserCollection" the Browser object's pendingDisconnect contains a Timeout object
The launcher sends the SIGKILL	no SIGKILL is sent
[coverage]: Writing coverage is present before onExit completed	no "Writing coverage" debug log before onExit completed
[reporter.junit]: JUnit results written	no "JUnit results written" log

With test coverage generated 👍

> ng test --no-watch --code-coverage

- Generating browser application bundles (phase: setup)...
  [config]: Loading config /home/vsts/work/1/s/Client/karma.conf.js
  [config]: autoWatch set to false, because of singleRun
.
.
.
INFO [coverage]: onBrowserComplete completed
DEBUG [launcher]: CAPTURED -> BEING_KILLED
DEBUG [launcher]: BEING_KILLED -> BEING_FORCE_KILLED
DEBUG [Chrome Headless 99.0.4844.51 (Linux x86_64)]: CONNECTED -> DISCONNECTED
TOTAL: 3231 SUCCESS

Chrome Headless 99.0.4844.51 (Linux x86_64): Executed 3231 of 3234 (skipped 3)�[32m SUCCESS�[39m (1 min 38.135 secs / 59.497 secs)
�[32mTOTAL: 3231 SUCCESS�[39m

10 03 2022 16:10:38.524:INFO [coverage]: onRunComplete called BrowserCollection {
browsers: [
Browser {
id: '25015465',
fullName: 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/99.0.4844.51 Safari/537.36',
name: 'Chrome Headless 99.0.4844.51 (Linux x86_64)',
lastResult: [BrowserResult],
disconnectsCount: 0,
activeSockets: [Array],
noActivityTimeout: 210000,
singleRun: true,
clientConfig: [Object],
collection: [BrowserCollection],
emitter: [Server],
socket: [Socket],
timer: [Object],
disconnectDelay: 210000,
log: [Logger],
noActivityTimeoutId: Timeout {
\_idleTimeout: 210000,
\_idlePrev: [TimersList],
\_idleNext: [Timeout],
\_idleStart: 278390,
\_onTimeout: [Function (anonymous)],
\_timerArgs: undefined,
\_repeat: null,
\_destroyed: false,
[Symbol(refed)]: true,
[Symbol(kHasPrimitive)]: false,
[Symbol(asyncId)]: 680445,
[Symbol(triggerId)]: 680443
},
pendingDisconnect: null,
state: 'DISCONNECTED'
}
],
emitter: EventEmitter {
\_events: [Object: null prototype] {},
\_eventsCount: 0,
\_maxListeners: undefined,
[Symbol(kCapture)]: false
}
}
INFO [coverage]: onRunComplete completed
DEBUG [karma-server]: Run complete, exiting.
DEBUG [launcher]: Disconnecting all browsers
DEBUG [launcher]: BEING_FORCE_KILLED -> BEING_FORCE_KILLED
DEBUG [proxy]: Destroying proxy agents
INFO [coverage]: onExit called

=============================== Coverage summary ===============================
Statements : 91.97% ( 23967/26059 )
Branches : 79.05% ( 9673/12237 )
Functions : 88.99% ( 6694/7522 )
Lines : 92.2% ( 22669/24586 )
================================================================================
WARN [launcher]: ChromeHeadless was not killed in 2000 ms, sending SIGKILL.
DEBUG [coverage]: Writing coverage to /home/vsts/work/1/s/Client/coverage
DEBUG [launcher]: Process ChromeHeadless exited with code null and signal SIGTERM
DEBUG [temp-dir]: Cleaning temp dir /tmp/karma-25015465
DEBUG [coverage]: Writing coverage to /home/vsts/work/1/s/Client/coverage
DEBUG [coverage]: Writing coverage to /home/vsts/work/1/s/Client/coverage
INFO [coverage]: onExit completed
DEBUG [reporter.junit]: JUnit results written to "/home/vsts/work/1/s/Client/TESTS-Chrome*Headless_99.0.4844.51*(Linux_x86_64).xml".
DEBUG [launcher]: Finished all browsers
DEBUG [launcher]: BEING_FORCE_KILLED -> FINISHED
DEBUG [launcher]: FINISHED -> FINISHED
DEBUG [karma-server]: Received stop event, exiting.
DEBUG [launcher]: Disconnecting all browsers
DEBUG [launcher]: FINISHED -> BEING_FORCE_KILLED
DEBUG [proxy]: Destroying proxy agents
INFO [coverage]: onExit called
INFO [coverage]: onExit completed ##[section]Finishing: Test

Without test coverage 👎

> ng test --no-watch --code-coverage

- Generating browser application bundles (phase: setup)...
  DEBUG [config]: Loading config /home/vsts/work/1/s/Client/karma.conf.js
  DEBUG [config]: autoWatch set to false, because of singleRun
.
.
.
DEBUG [launcher]: CAPTURED -> BEING_KILLED
DEBUG [launcher]: BEING_KILLED -> BEING_FORCE_KILLED
DEBUG [Chrome Headless 99.0.4844.51 (Linux x86_64)]: CONNECTED -> DISCONNECTED
TOTAL: 3231 SUCCESS

Chrome Headless 99.0.4844.51 (Linux x86_64): Executed 3231 of 0 (skipped 3)�[32m SUCCESS�[39m (2 mins 15.127 secs / 59.651 secs)
�[32mTOTAL: 3231 SUCCESS�[39m

INFO [coverage]: onRunComplete called BrowserCollection {
browsers: [
Browser {
id: '13380493',
fullName: 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/99.0.4844.51 Safari/537.36',
name: 'Chrome Headless 99.0.4844.51 (Linux x86_64)',
lastResult: [BrowserResult],
disconnectsCount: 0,
activeSockets: [Array],
noActivityTimeout: 210000,
singleRun: true,
clientConfig: [Object],
collection: [BrowserCollection],
emitter: [Server],
socket: [Socket],
timer: [Object],
disconnectDelay: 210000,
log: [Logger],
noActivityTimeoutId: Timeout {
\_idleTimeout: 210000,
\_idlePrev: [TimersList],
\_idleNext: [Timeout],
\_idleStart: 285367,
\_onTimeout: [Function (anonymous)],
\_timerArgs: undefined,
\_repeat: null,
\_destroyed: false,
[Symbol(refed)]: true,
[Symbol(kHasPrimitive)]: false,
[Symbol(asyncId)]: 681197,
[Symbol(triggerId)]: 681196
},
pendingDisconnect: Timeout {
\_idleTimeout: -1,
\_idlePrev: null,
\_idleNext: null,
\_idleStart: 187011,
\_onTimeout: null,
\_timerArgs: undefined,
\_repeat: null,
\_destroyed: true,
[Symbol(refed)]: true,
[Symbol(kHasPrimitive)]: false,
[Symbol(asyncId)]: 646232,
[Symbol(triggerId)]: 646172
},
state: 'DISCONNECTED'
}
],
emitter: EventEmitter {
\_events: [Object: null prototype] {},
\_eventsCount: 0,
\_maxListeners: undefined,
[Symbol(kCapture)]: false
}
}
INFO [coverage]: onRunComplete completed
DEBUG [karma-server]: Run complete, exiting.
DEBUG [launcher]: Disconnecting all browsers
DEBUG [launcher]: BEING_FORCE_KILLED -> BEING_FORCE_KILLED
DEBUG [proxy]: Destroying proxy agents
INFO [coverage]: onExit called
INFO [coverage]: onExit completed
DEBUG [launcher]: Process ChromeHeadless exited with code null and signal SIGTERM
DEBUG [temp-dir]: Cleaning temp dir /tmp/karma-13380493
DEBUG [launcher]: Finished all browsers
DEBUG [launcher]: BEING_FORCE_KILLED -> FINISHED
DEBUG [launcher]: FINISHED -> FINISHED
DEBUG [karma-server]: Received stop event, exiting.
DEBUG [launcher]: Disconnecting all browsers
DEBUG [launcher]: FINISHED -> BEING_FORCE_KILLED
DEBUG [proxy]: Destroying proxy agents
INFO [coverage]: onExit called
INFO [coverage]: onExit completed ##[section]Finishing: Test

[devoto13]

Thanks for the details @ozkoidi! I don't think pendingDisconnect is relevant, it probably happened sometime during the test execution and karma has recovered from it given that the timer is inactive (_idleTimeout: -1).

Missing onBrowserComplete called looks relevant though. junit reporter is affected by this as well. Unfortunately, I was not able to figure out how exactly could it happen from reading the code... There is a relatively sync chain of browser_complete -> browser_complete_with_no_more_retries -> run_complete -> Karma shut down, but I don't understand where it could race, so that browser_complete event is not reaching plugins at all. Without the process.kill the pending event listeners should prevent Node process from exiting and they should be called even if out of order 🤔

Can you please use the two below branches and share logs for the two cases to further investigate this problem?

$ npm i devoto13/karma#test-coverage-1
$ npm i devoto13/karma-coverate#test-coverage-1

If possible can you also attach unedited logs as files as timestamps may be useful in figuring out what is going on?

[ozkoidi]

Hello again @devoto13, I tested your latest branches in the pipelines and I get the same behavior.
I attach here a zip file with the full unit tests log output (Karma's logLevel set to LOG_DEBUG).
The zip contains 2 text files (coverage-devoto-1.txt and no-coverage-devoto-1.txt). For confidentiality reasons I had to rename the tests but I don't think that should be an impediment.

I hope you find these logs useful to find the problem, thanks for your time.

full-logs-devoto-1.zip

[devoto13]

Thank you for the help debuggin it, @ozkoidi! Are you sure you were running the code from test-coverage-1 branches though? I don't see any of the logs added in devoto13/karma@727469e for example.

And specific test names are irrelevant, I was primarily interested to look into the timing of the newly added logs.

[ozkoidi]

Yes, I even uninstalled and re-installed karma and karma-coverage again today to be sure. You can see it in this extract from my package-lock.json:

 "karma": {
      "version": "git+ssh://git@github.com/devoto13/karma.git#c411e999b433a1c135305953658f9662571312a4",
      "dev": true,
      "from": "karma@github:devoto13/karma#test-coverage-1",
   ...
 "karma-coverage": {
      "version": "git+ssh://git@github.com/devoto13/karma-coverage.git#a7f67f27edd9ea515bfcf39e6fef2259f89360c7",
      "dev": true,
      "from": "karma-coverage@github:devoto13/karma-coverage#test-coverage-1",
   ...

I run the pipeline several times and searched for the logs you added but indeed, they never show up on the output.
If there is anything else I could do to help please let me know as this issue blocks our pipelines often.

[devoto13]

Okey, thanks @ozkoidi! So it seems that the browser_complete event is not coming from the browser over the socket.io connection, that's interesting. I'll look more into it and get back to you if I need more information.

Thanks again for your help in debugging this.

[danemesis]

I had the very same issue and was trying for weeks now to solve it. I was able constantly solved it with adding to karma.config.js following 2 options

browserSocketTimeout: 8 * 60 * 40000,
pingTimeout: 8 * 60 * 40000,

so the resulted file is

// Karma configuration file, see link for more information
// https://karma-runner.github.io/1.0/config/configuration-file.html
const isDocker = require('is-docker');

module.exports = (config) => {
	config.set({
		basePath: '',
		frameworks: ['jasmine', '@angular-devkit/build-angular'],
		plugins: [
			require('karma-jasmine'),
			require('karma-chrome-launcher'),
			require('karma-jasmine-html-reporter'),
			require('karma-junit-reporter'),
			require('karma-coverage'),
			require('@angular-devkit/build-angular/plugins/karma'),
		],
		client: {
			jasmine: {
				random: false,
				stopOnFailure: true,
				failFast: true,
			},
			clearContext: false, // leave Jasmine Spec Runner output visible in browser
		},
		junitReporter: {
			outputDir: './reports',
			outputFile: 'karma.xml',
			useBrowserName: false,
		},
		coverageReporter: {
			dir: './reports/coverage',
			subdir: '.',
			includeAllSources: true,
			reporters: [
				{ type: 'cobertura' },
				{ type: 'lcovonly' },
				{ type: 'html' },
				{ type: 'text' },
			],
			check: {
				// thresholds for all files
				global: {
					statements: 76.25,
					branches: 55.95,
					functions: 71.4,
					lines: 75.8,
				}
			},
		},
		reporters: ['progress', 'kjhtml', 'junit', 'coverage'],
		port: 9876,
		colors: true,
		logLevel: config.LOG_INFO,
		autoWatch: true,
		// to avoid DISCONNECTED messages
		browserDisconnectTimeout: 400000, // default 2000
		browserDisconnectTolerance: 6, // default 0
		browserNoActivityTimeout: 8 * 60 * 40000, //default 10000
		captureTimeout: 8 * 60 * 40000, //default 60000
		browserSocketTimeout: 8 * 60 * 40000,
		pingTimeout: 8 * 60 * 40000,
		browsers: ['ChromeCustom'],
		singleRun: false,
		customLaunchers: {
			ChromeCustom: {
				base: 'ChromeHeadless',
				flags: isDocker() ? ['--disable-gpu', '--no-sandbox'] : [],
			},
		},
	});
};

[ozkoidi]

Thank you so much @danbilokha! After adding browserSocketTimeout and pingTimeout it hasn't failed a single time so far 🎉

I already had big values for browserDisconnectTimeout, browserNoActivityTimeout and played with browserDisconnectTolerance in the past but didn't help. However adding the properties you mentioned made the coverage reports to always be generated 👍

Even though is not an actual fix, it does unblock our pipelines, so thanks again for your suggestion :)

[mlakmal]

had same issue while running unit tests through linux docker container with low cpu specs. using browserSocketTimeout and pingTimeout worked!!!

for anyone searching like me below are our app configs.
angular v14
with nrwl nx v14
using karma