Code Coverage

[denis-yuen]

I was wondering if Cypress had a recommended tool for calculating code coverage? and if so are there any examples?

Following up on cypress-io/cypress-example-recipes#3 and opening an issue here for discussion as recommended.

For context, we previously used Protractor paired with Istanbul ( https://github.com/gotwarlost/istanbul ) to calculate code coverage for our application. We were wondering if any one has experience with something similar when working with Cypress

FYI @agduncan94

[brian-mann]

There is nothing currently built into Cypress to do this. Adding code coverage around e2e tests is much harder than unit and its possible it may not be feasible to do in a generic way.

If you can supply the processes / code you used to add coverage via Protractor that would be helpful.

The problem is that your typical JS src files can be written in many different ways, ES6, Typescript, Coffeescript, and generally the final JS files are transpiled, and then bundled via Browserify or Webpack.

It is possible to add code coverage to JS files served by a webserver though, so it's something we might be able to solve. IE: using something like this: https://github.com/gotwarlost/istanbul-middleware

But in doing so, the problem is that it would have to be integrated at your server level, and is specific to each way you build your src files - aka each individual application.

It may be possible for Cypress to do this at the network proxy level, but it would unlikely to be easy.

[agduncan94]

In our application we actually used Karma along with Istanbul (karma-coverage) to calculate code coverage on our unit tests.

We setup a Grunt task which uses the karma-coverage plugin to generate a coverage report on the unit tests. We have a karma config file similar to the one in the karma-coverage link above where we specify which JavaScript files to test.

What we were hoping to do as an alternate to karma and karma-coverage is to just call Istanbul and Mocha directly (or use something else if more appropriate) on the tests. This does not work because cypress has some special variables (like cy) which are not recognized by Mocha.

This is an example of how we would expect to call Istanbul and mocha if we were just using mocha directly, instead of cypress.

So I think we are most interested in coverage for unit testing.

[brian-mann]

@agduncan94 thanks for that explanation.

To clarify - doing unit testing when it comes to code coverage is fairly straightforward and absolutely can be done.

To date Cypress has been primarily used as integration and e2e testing whereby code coverage is substantially different.

With 0.18.0 we are now seeing users begin to add in unit testing and I have described the updates we need to make to Cypress to make this a first class feature..

[mike-engel]

This might be a bit naïve, but would it be possible to use the new code coverage tool in the chrome dev tools to accomplish this? Admittedly, I haven't looked into it much, and it would only work in chrome for now. https://developers.google.com/web/updates/2017/04/devtools-release-notes#coverage

[pvdyck]

There should be a way to instrument the tests to extract coverage and even export it to https://coveralls.io/ ( lemurheavy/coveralls-public#1023 )

Any chance to see this happening ? What do we need ?

[brian-mann]

Cypress is open source, so you're welcome to take a look around to see what we'd need to do.

Code coverage during e2e tests is completely different than unit tests. There is no dependency tree, there are 3rd party libs, and there is no access to the original source (even with source maps) due to the browser build process that every JS codebase undergoes.

Cypress could theoretically instrument the code for you at the network layer, as its headed for the browser. But this has substantial challenges - it would be unidirectional, and we'd have no way of mapping the underlying built JS code back to the original (even with source maps its not enough).

So while we could provide code coverage on the final built JS files, it would be mostly garbage from babel, typescript, or whatever else you're using to transpile. It would also include all of the 3rd party code you import in.

All in all, not very useful.

We'd likely need to expose and build an API on top of Cypress's backend process that your server could send the original source files which we could map against the transpiled versions to come up with a lightweight set of instrumented files. We could then use those to determine the standard code coverage.

To be clear - this isn't a challenge as long as the code is instrumented properly, Cypress like every other tool could easily generate code coverage. The problem is that during e2e tests we receive the final built JS code where trying to go backwards to instrument is not really going to work.

Another option is that your own webserver could send the instrumented code (that you'd have to manage yourself). After you do that, it would be fairly trivial to have Cypress be able to read off the instrumented reports after test execution and then aggregate the results.

Lot of potential directions to go here, and is likely a good bit of work to champion something that's generic enough to work in most situations but easy enough so people could drop it in without writing a lot of code for their unique situation.

[denis-yuen]

Thanks for the explanation BTW!

[bahmutov]

@denis-yuen are you trying to compute code coverage for your tests (they should be executing all lines, so 100% right away...) or production code? If production code, maybe stick https://github.com/bahmutov/was-tested between server and Cypress and get code coverage from there - but we do not think code coverage is helpful in production E2E tests.

We are thinking about a different type of coverage - UI element coverage. You are probably more interested in knowing if all buttons or menu elements on the page were exercised by the Cypress end to end tests, rather than all code.

[atfzl]

I have set up coverage for Cypress if anyone wants to try.

First you need to instrument your code, for this I have used istanbul-instrumenter-loader with webpack.

Then add this code in your <project>/cypress/support/index.js

const istanbul = require('istanbul-lib-coverage');

const map = istanbul.createCoverageMap({});

Cypress.on('window:before:unload', e => {
  const coverage = e.currentTarget.__coverage__;

  if (coverage) {
    map.merge(coverage);
  }
});

after(() => {
  cy.window().then(win => {
    const coverage = win.__coverage__;

    if (coverage) {
      map.merge(coverage);
    }

    cy.writeFile('.nyc_output/out.json', JSON.stringify(map));
    cy.exec('nyc report --reporter=html');
  });
});

Here 'window:before:unload' event is used for capturing coverage which would be lost after a cy.visit() or refresh etc.

you also need to install nyc to generate reports which will be saved in <project>/coverage folder.

[bahmutov]

NICE @atfzl !!!

[9fingerdev]

Been looking at this but so far have not quite figured out how to get things working. Getting an out.json file but its empty. @atfzl can you shed some more light on what steps you took to config istanbul-instrumenter-loader?

[atfzl]

@9fingerdev this is where i used the istanbul-istrumenter-loader:

rules: [
      {
        test: /\.(jsx|js|tsx|ts)$/,
        include: path.resolve(__dirname, '../src'),
        rules: (isDebug
          ? [
              {
                loader: 'istanbul-instrumenter-loader',
                options: {
                  esModules: true,
                },
              },
            ]
          : []
        ).concat([{ loader: 'awesome-typescript-loader' }]),
      },

I'll create a recipe project on weekend showing complete process.

[9fingerdev]

@atfzl any progress on that recipe project? We are using Mithril.js + WebPack and keep running into problems.