Code Coverage

[denis-yuen]

I was wondering if Cypress had a recommended tool for calculating code coverage? and if so are there any examples?

Following up on cypress-io/cypress-example-recipes#3 and opening an issue here for discussion as recommended.

For context, we previously used Protractor paired with Istanbul ( https://github.com/gotwarlost/istanbul ) to calculate code coverage for our application. We were wondering if any one has experience with something similar when working with Cypress

FYI @agduncan94

[brian-mann]

There is nothing currently built into Cypress to do this. Adding code coverage around e2e tests is much harder than unit and its possible it may not be feasible to do in a generic way.

If you can supply the processes / code you used to add coverage via Protractor that would be helpful.

The problem is that your typical JS src files can be written in many different ways, ES6, Typescript, Coffeescript, and generally the final JS files are transpiled, and then bundled via Browserify or Webpack.

It is possible to add code coverage to JS files served by a webserver though, so it's something we might be able to solve. IE: using something like this: https://github.com/gotwarlost/istanbul-middleware

But in doing so, the problem is that it would have to be integrated at your server level, and is specific to each way you build your src files - aka each individual application.

It may be possible for Cypress to do this at the network proxy level, but it would unlikely to be easy.

[agduncan94]

In our application we actually used Karma along with Istanbul (karma-coverage) to calculate code coverage on our unit tests.

We setup a Grunt task which uses the karma-coverage plugin to generate a coverage report on the unit tests. We have a karma config file similar to the one in the karma-coverage link above where we specify which JavaScript files to test.

What we were hoping to do as an alternate to karma and karma-coverage is to just call Istanbul and Mocha directly (or use something else if more appropriate) on the tests. This does not work because cypress has some special variables (like cy) which are not recognized by Mocha.

This is an example of how we would expect to call Istanbul and mocha if we were just using mocha directly, instead of cypress.

So I think we are most interested in coverage for unit testing.

[brian-mann]

@agduncan94 thanks for that explanation.

To clarify - doing unit testing when it comes to code coverage is fairly straightforward and absolutely can be done.

To date Cypress has been primarily used as integration and e2e testing whereby code coverage is substantially different.

With 0.18.0 we are now seeing users begin to add in unit testing and I have described the updates we need to make to Cypress to make this a first class feature..

[mike-engel]

This might be a bit naïve, but would it be possible to use the new code coverage tool in the chrome dev tools to accomplish this? Admittedly, I haven't looked into it much, and it would only work in chrome for now. https://developers.google.com/web/updates/2017/04/devtools-release-notes#coverage

[pvdyck]

There should be a way to instrument the tests to extract coverage and even export it to https://coveralls.io/ ( lemurheavy/coveralls-public#1023 )

Any chance to see this happening ? What do we need ?

[brian-mann]

Cypress is open source, so you're welcome to take a look around to see what we'd need to do.

Code coverage during e2e tests is completely different than unit tests. There is no dependency tree, there are 3rd party libs, and there is no access to the original source (even with source maps) due to the browser build process that every JS codebase undergoes.

Cypress could theoretically instrument the code for you at the network layer, as its headed for the browser. But this has substantial challenges - it would be unidirectional, and we'd have no way of mapping the underlying built JS code back to the original (even with source maps its not enough).

So while we could provide code coverage on the final built JS files, it would be mostly garbage from babel, typescript, or whatever else you're using to transpile. It would also include all of the 3rd party code you import in.

All in all, not very useful.

We'd likely need to expose and build an API on top of Cypress's backend process that your server could send the original source files which we could map against the transpiled versions to come up with a lightweight set of instrumented files. We could then use those to determine the standard code coverage.

To be clear - this isn't a challenge as long as the code is instrumented properly, Cypress like every other tool could easily generate code coverage. The problem is that during e2e tests we receive the final built JS code where trying to go backwards to instrument is not really going to work.

Another option is that your own webserver could send the instrumented code (that you'd have to manage yourself). After you do that, it would be fairly trivial to have Cypress be able to read off the instrumented reports after test execution and then aggregate the results.

Lot of potential directions to go here, and is likely a good bit of work to champion something that's generic enough to work in most situations but easy enough so people could drop it in without writing a lot of code for their unique situation.

[denis-yuen]

Thanks for the explanation BTW!

[bahmutov]

@denis-yuen are you trying to compute code coverage for your tests (they should be executing all lines, so 100% right away...) or production code? If production code, maybe stick https://github.com/bahmutov/was-tested between server and Cypress and get code coverage from there - but we do not think code coverage is helpful in production E2E tests.

We are thinking about a different type of coverage - UI element coverage. You are probably more interested in knowing if all buttons or menu elements on the page were exercised by the Cypress end to end tests, rather than all code.

[atfzl]

I have set up coverage for Cypress if anyone wants to try.

First you need to instrument your code, for this I have used istanbul-instrumenter-loader with webpack.

Then add this code in your <project>/cypress/support/index.js

const istanbul = require('istanbul-lib-coverage');

const map = istanbul.createCoverageMap({});

Cypress.on('window:before:unload', e => {
  const coverage = e.currentTarget.__coverage__;

  if (coverage) {
    map.merge(coverage);
  }
});

after(() => {
  cy.window().then(win => {
    const coverage = win.__coverage__;

    if (coverage) {
      map.merge(coverage);
    }

    cy.writeFile('.nyc_output/out.json', JSON.stringify(map));
    cy.exec('nyc report --reporter=html');
  });
});

Here 'window:before:unload' event is used for capturing coverage which would be lost after a cy.visit() or refresh etc.

you also need to install nyc to generate reports which will be saved in <project>/coverage folder.

[bahmutov]

NICE @atfzl !!!

[9fingerdev]

Been looking at this but so far have not quite figured out how to get things working. Getting an out.json file but its empty. @atfzl can you shed some more light on what steps you took to config istanbul-instrumenter-loader?

[atfzl]

@9fingerdev this is where i used the istanbul-istrumenter-loader:

rules: [
      {
        test: /\.(jsx|js|tsx|ts)$/,
        include: path.resolve(__dirname, '../src'),
        rules: (isDebug
          ? [
              {
                loader: 'istanbul-instrumenter-loader',
                options: {
                  esModules: true,
                },
              },
            ]
          : []
        ).concat([{ loader: 'awesome-typescript-loader' }]),
      },

I'll create a recipe project on weekend showing complete process.

[9fingerdev]

@atfzl any progress on that recipe project? We are using Mithril.js + WebPack and keep running into problems.

[atfzl]

@9fingerdev, project showcasing cypress coverage: https://github.com/atfzl/react-redux-typescript-boilerplate

[atfzl]

For some reason istanbul-instrumenter-loader does not work with babel. For working with babel you can try babel-plugin-istanbul

[atfzl]

To instrument code with babel, there is no need to change the webpack config, just add the istanbul plugin.
sample .babelrc for working instrumented code:

{
  "presets": [
    "react",
    "flow",
    [
      "es2015",
      {
        "modules": false
      }
    ],
    "stage-2"
  ],
  "plugins": [
    "transform-runtime",
    "lodash",
    "transform-decorators-legacy",
    "istanbul"
  ],
  "env": {
    "test": {
      "plugins": [
        "transform-es2015-modules-commonjs"
      ]
    }
  },
  "retainLines": true
}

[gkemp94]

@atfzl This is exactly what I was looking for. We're using gulp though. Would the configuration be similar?

[atfzl]

@gkemp94 if you are using babel then you only need to add plugin in .babelrc, should work with gulp as well.

[abataub]

@atfzl Can you please highlight how can I configure it with grunt? I'm not using webpack at all. Thanks.

[robinyy]

Instrumenting the codes will do ONLY with non-compressed JS. However, a typical process of shipping the normal web application would be: Linting & UT passed (source code or transpiled) -> Packaging (minified & mangled) -> E2E/Integration testing (against compressed version of JS) -> promote to production

It would be difficult to calculate the coverage in the above pipeline unless we create a separated one only for the E2E coverage

[abataub]

@atfzl Can you please show light on to find code coverage when running tests in isolation mode?
cypress run --spec cypress/integration/*

Cypress version: 3.0.1

[atfzl]

@abataub the steps i have mentioned also works in isolation mode. Let me know what is the exact issue.
And for making it work with grunt see the comment at #346 (comment) if you are using babel.

[abataub]

Thanks, @atfzl. I just made changes and it works.

[jackjocross]

Thanks @atfzl that was super helpful!

In order to end up with cumulative coverage across multiple cypress/integration/*.js files, I had to add some code to read and merge any existing '.nyc_output/out.json file. Not sure if anyone else ran into the same issue.

[afenton90]

@atfzl have you tried this with env babel plugin? I can't get it to work in my setup.
How sensitive is the babel example you have given?

[paulfalgout]

The v3 solution I got working.. it's probably overkilling it in some form, but this allowed for the node stored map to be reused across all of the tests merging them along the way. I set the env coverage to open when running cypress open and it only runs the report for whichever file I run.

In plugins:

    if(config.env.coverage) {
        const istanbul = require('istanbul-lib-coverage');
        coverageMap = istanbul.createCoverageMap({});

        on('task', {
            'coverage'(coverage) {
                coverageMap.merge(coverage);
                return JSON.stringify(coverageMap);
            }
        });
    }

In support:

if(Cypress.env('coverage')) {
    afterEach(function() {
        const coverageFile = `${ Cypress.config('coverageFolder') }/out.json`;

        cy.window().then(win => {
            const coverage = win.__coverage__;

            if(!coverage) return;

            cy.task('coverage', coverage).then(map => {
                cy.writeFile(coverageFile, map);

                if(Cypress.env('coverage') === 'open') {
                    cy.exec('nyc report --reporter=html');
                }
            });
        });
    });
}

My npm script is essentially:
cross-env NODE_ENV=test yarn run build && cypress run --env coverage=true && nyc report --reporter=html --reporter=text

[bahmutov]

This week I have made a presentation at Øredev showing what we think is a good replacement for code coverage for end-to-end tests. Look at the slides here https://slides.com/bahmutov/well-tested-software and in particular read the following blog posts:

• Element coverage for end-to-end tests
• state coverage in blog post Hyperapp state machine web app

I think both element coverage and state machine coverage are a lot more meaningful than code statement coverage, and we will develop these ideas more in the future.

[frattaro]

@bahmutov i assume state coverage would need a lib for each state management solution? Vuex, redux, etc.

[bahmutov]

Even worse/better - it refers to modeling application as a state transition machines, something like “xstate” library for example

…

Sent from my iPhone

On Nov 22, 2018, at 18:22, Anton Frattaroli ***@***.***> wrote: @bahmutov i assume state coverage would need a lib for each state management solution? Vuex, redux, etc. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[Vandivier]

@rwwagner90 merging mocha and cypress coverage - nice idea. I notice the cited repo now uses Jest. Is that a recent change? What was the reason for using Jest instead of Cypress for UI component testing? If you're not using Cypress for UI component testing does then what coverage are you merging?

[RobbieTheWagner]

@Vandivier we're no longer merging coverage. We're just using Jest and it does its own coverage. We no longer try to get coverage out of Cypress.

[ifiokjr]

@rwwagner90 interested to know if there's any particular reason for not getting coverage from Cypress. I've contemplated putting something together. Is it not worth the effort?

[RobbieTheWagner]

@ifiokjr we decided that ultimately, Cypress coverage was not a good representation of real coverage. You may "cover" 80% of lines just by booting the app up and visiting a page, but you're really only testing the functionality of one function. We decided coverage only made sense from unit tests, and we would wait for Cypress to expose more meaningful metrics like element coverage or application state coverage, as they mention here https://docs.cypress.io/faq/questions/general-questions-faq.html#Is-there-code-coverage

[frattaro]

Unit testing with cypress is a thing that can be done. Info doesn't directly impact the conversation but is something to consider.

[paulfalgout]

I find the coverage reports I can get from Cypress to be quite useful. Yes it doesn't tell you what is covered or not as false positives are high, but it can tell you what is definitely not covered. In some cases I discovered tests we thought were covering the use case, but weren't. However I generate coverage reports two ways. I merge a report across the entire code base when run headlessly, but I also run coverage tests isolated on particular tests from the gui. The coverage reports help me write better tests and can in some instances help catch dead code. They don't do the same job as unit test coverage, but it's useful data none-the-less.

[papb]

I am using TypeScript, I tried to adapt the code by @paulfalgout to make it work with TypeScript but got stuck. Thankfully someone kindly helped me in my question on stackoverflow, so if anyone is trying to make Cypress + TypeScript + IstanbulJS work together, it can be done as well, just check the accepted answer 😬

[bahmutov]

Hi everyone, I have blogged how to instrument application code and report code coverage from end-to-end tests:

• first read the blog post "https://glebbahmutov.com/blog/code-coverage-for-e2e-tests/" where instrumentation is done as a separate build step
• demo repo https://github.com/bahmutov/demo-battery-api branch coverage-step

Then read the following posts:

• "Code Coverage by Parcel Bundler" which shows how to instrument the application on the fly using babel-plugin-istanbul

• "Combined End-to-end and Unit Test Coverage" where both application and unit tests are instrumented using same code and the combined coverage report is saved - easily getting to 100% code coverage

Recommended plugin: https://github.com/cypress-io/cypress-istanbul for merging code coverage information into correct object

[Stephen2]

THANK YOU so much @bahmutov

[bahmutov]

Closing this issue, as we have released plugin and detailed documentation how to get end-to-end, unit and full-stack code coverage

• read our Code Coverage guide
• use @cypress/code-coverage plugin