nyc config `all:true` does not work

[penx]

Follow up to #194

Logs and screenshots

✗ DEBUG=code-coverage yarn coverage:integration 
yarn run v1.21.1
$ rm -rf coverage/integration && mkdir -p coverage/integration && yarn cypress run
$ /Users/penx/Development/storybook-code-coverage/node_modules/.bin/cypress run
GVA encoder info: AMD performance mode : 2 
GVA encoder info: deleteSCDMetalContext : texture cache hits: 0, misses: 0

====================================================================================================

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:    4.4.0                                                                              │
  │ Browser:    Electron 80 (headless)                                                             │
  │ Specs:      1 found (spec.js)                                                                  │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


────────────────────────────────────────────────────────────────────────────────────────────────────
                                                                                                    
  Running:  spec.js                                                                         (1 of 1)


  ✓ has a visible header (381ms)
  code-coverage parsed sent coverage +0ms
  code-coverage wrote coverage file /Users/penx/Development/storybook-code-coverage/.nyc_output/out.json +2ms
  code-coverage NYC file /Users/penx/Development/storybook-code-coverage/.nyc_output/out.json has 5 key(s) +0ms
  code-coverage 1 key /Users/penx/Development/storybook-code-coverage/src/utilities/useToggle.js file path /Users/penx/Development/storybook-code-coverage/src/utilities/useToggle.js +0ms
  code-coverage 2 key /Users/penx/Development/storybook-code-coverage/src/design-system/Toggle/index.js file path /Users/penx/Development/storybook-code-coverage/src/design-system/Toggle/index.js +0ms
  code-coverage 3 key /Users/penx/Development/storybook-code-coverage/src/design-system/GlobalStyles/index.js file path /Users/penx/Development/storybook-code-coverage/src/design-system/GlobalStyles/index.js +0ms
  code-coverage in file /Users/penx/Development/storybook-code-coverage/.nyc_output/out.json all files are not found? false +0ms
  code-coverage NYC file /Users/penx/Development/storybook-code-coverage/.nyc_output/out.json has 5 key(s) +2ms
  code-coverage combined NYC options { 'report-dir': './coverage', reporter: [ 'lcov', 'clover', 'json' ], extension: [ '.js', '.cjs', '.mjs', '.ts', '.tsx', '.jsx' ], excludeAfterRemap: true, all: true, include: [ 'src/**/*.js' ], exclude: [ '**/*.test.js', '**/test.js', '**/*.stories.js', '**/stories.js' ] } +0ms
  code-coverage calling NYC reporter with options { 'report-dir': '/Users/penx/Development/storybook-code-coverage/coverage', reporter: [ 'lcov', 'clover', 'json' ], extension: [ '.js', '.cjs', '.mjs', '.ts', '.tsx', '.jsx' ], excludeAfterRemap: true, all: true, include: [ 'src/**/*.js' ], exclude: [ '**/*.test.js', '**/test.js', '**/*.stories.js', '**/stories.js' ], tempDir: '/Users/penx/Development/storybook-code-coverage/.nyc_output' } +28ms
  code-coverage current working directory is /Users/penx/Development/storybook-code-coverage +0ms
  code-coverage after reporting, returning the report folder name /Users/penx/Development/storybook-code-coverage/coverage +56ms
  code-coverage Final coverage in /Users/penx/Development/storybook-code-coverage/coverage/coverage-final.json +0ms
  code-coverage ✅ /Users/penx/Development/storybook-code-coverage/src/index.js statements covered 1/1 +0ms
  code-coverage ✅ /Users/penx/Development/storybook-code-coverage/src/application/App.js statements covered 2/2 +0ms
  code-coverage ✅ /Users/penx/Development/storybook-code-coverage/src/design-system/GlobalStyles/index.js statements covered 7/7 +0ms
  code-coverage ✅ /Users/penx/Development/storybook-code-coverage/src/design-system/Toggle/index.js statements covered 5/5 +0ms
  code-coverage ⚠️ /Users/penx/Development/storybook-code-coverage/src/utilities/useToggle.js statements covered 6/10 +1ms

  1 passing (505ms)


  (Results)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Tests:        1                                                                                │
  │ Passing:      1                                                                                │
  │ Failing:      0                                                                                │
  │ Pending:      0                                                                                │
  │ Skipped:      0                                                                                │
  │ Screenshots:  0                                                                                │
  │ Video:        true                                                                             │
  │ Duration:     0 seconds                                                                        │
  │ Spec Ran:     spec.js                                                                          │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


  (Video)

  -  Started processing:  Compressing to 32 CRF                                                     
  -  Finished processing: /Users/penx/Development/storybook-code-coverage/cypress/vid    (0 seconds)
                          eos/spec.js.mp4                                                           


====================================================================================================

  (Run Finished)


       Spec                                              Tests  Passing  Failing  Pending  Skipped  
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔  spec.js                                  497ms        1        1        -        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✔  All specs passed!                        497ms        1        1        -        -        -  

✨  Done in 8.89s.

Versions

• What is this plugin's version?
  3.4.1

• What is Cypress version?
  4.4.0

• What is your operating system?
  Mac OS X 10.15.4

• What is the shell? zsh

• What is the Node version?
  12.14.1

• What is the NPM version?
  6.13.4

• How do you instrument your application? Cypress does not instrument web application code, so you need to do it yourself.

• When running tests, if you open the web application in regular browser, and open DevTools, do you see window.__coverage__ object? Can you paste a screenshot?

Yes

• Is there .nyc_output folder? Is there .nyc_output/out.json file. Is it empty? Can you paste at least part of it so we can see the keys and file paths?

Yes

• Do you have any custom NYC settings in package.json (nyc object) or in other NYC config files

Yes, as per 'describe the bug' below

• Do you run Cypress tests in a Docker container?

No

Describe the bug
If I have the following in my package.json:

  "nyc": {
    "all": true,
    "include": ["src/**/*.js"],
    "exclude": ["**/*.test.js", "**/*.stories.js"]
  }

I would expect to see files that have 0% coverage in the report, but they don't appear.

Link to the repo

https://github.com/penx/storybook-code-coverage

^ at present, you will need to update this repo to 3.4.1 and remove "report-dir": "coverage/integration", from package.json (as this setting seems to be bugged in 3.4.1)

[penx]

I'm thinking this could be an issue with @cypress-io/instrument-cra

https://github.com/cypress-io/instrument-cra/blob/bf977b19c77799e8b9c9f4a4ee06ae42fe7678ac/index.js#L33

It looks like instrument-cra pushes the babel plugin istanbul plugin in to the webpack build, so perhaps Istanbul is only aware of files that are being built by webpack. In the example above, I have files in my project that aren't included in the webpack build, but I would want these to be included in the code coverage report (it is useful to utilise code coverage reports from integration tests to identify redundant files that are candidates for deletion).

[bahmutov]

Hi, I looked at the project

• I have fixed the report dir, seems NYC really was using the camelCased report dir option, fixed in https://github.com/cypress-io/code-coverage/releases/tag/v3.4.2
• Hmm, regarding missing files in src/utilities - I don't think NYC itself does anything differently. If after Cypress test I run

--------------------------------|---------|----------|---------|---------|-------------------
File                            | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
--------------------------------|---------|----------|---------|---------|-------------------
All files                       |      84 |    71.43 |   66.67 |     100 |                   
 src                            |     100 |      100 |     100 |     100 |                   
  index.js                      |     100 |      100 |     100 |     100 |                   
 src/application                |     100 |      100 |     100 |     100 |                   
  App.js                        |     100 |      100 |     100 |     100 |                   
 src/design-system/GlobalStyles |     100 |       50 |     100 |     100 |                   
  index.js                      |     100 |       50 |     100 |     100 | 21                
 src/design-system/Toggle       |     100 |       75 |     100 |     100 |                   
  index.js                      |     100 |       75 |     100 |     100 | 12                
 src/utilities                  |      60 |      100 |      20 |     100 |                   
  useToggle.js                  |      60 |      100 |      20 |     100 |                   
--------------------------------|---------|----------|---------|---------|-------------------

I don't see additional files in the list. Have you seen empty files before?

• for your known issue Can't configure cypress-specific nyc settings outside of package.json - I have closed this in https://github.com/cypress-io/code-coverage/releases/tag/v3.4.0

[penx]

I don't see additional files in the list. Have you seen empty files before?

I haven't seen Cypress report on uncovered files before, if that's what you mean? But I've only just started with this.

However, if you run yarn coverage:unit or coverage:visual-regression on the above project you'll see that Istanbul will report on uncovered files:

--------------------------------|----------|----------|----------|----------|-------------------|
File                            |  % Stmts | % Branch |  % Funcs |  % Lines | Uncovered Line #s |
--------------------------------|----------|----------|----------|----------|-------------------|
All files                       |    32.26 |    14.29 |    33.33 |       25 |                   |
 src                            |        0 |      100 |      100 |        0 |                   |
  index.js                      |        0 |      100 |      100 |        0 |                 6 |
 src/application                |        0 |      100 |        0 |        0 |                   |
  App.js                        |        0 |      100 |        0 |        0 |               7,8 |
 src/design-system/Alert        |        0 |      100 |        0 |        0 |                   |
  index.js                      |        0 |      100 |        0 |        0 |                 3 |
 src/design-system/GlobalStyles |        0 |        0 |        0 |        0 |                   |
  index.js                      |        0 |        0 |        0 |        0 |4,7,12,14,15,19,20 |
 src/design-system/Message      |        0 |      100 |        0 |        0 |                   |
  index.js                      |        0 |      100 |        0 |        0 |                 3 |
 src/design-system/Toggle       |        0 |        0 |        0 |        0 |                   |
  index.js                      |        0 |        0 |        0 |        0 |        4,5,6,9,10 |
 src/utilities                  |    83.33 |      100 |    83.33 |    85.71 |                   |
  addOne.js                     |        0 |      100 |        0 |        0 |                 1 |
  useToggle.js                  |      100 |      100 |      100 |      100 |                   |
--------------------------------|----------|----------|----------|----------|-------------------|

[bahmutov]

Yeah, I think you have identified the problem - the babel-plugin-istanbul instrumet-CRA is inserting is only instrumenting the files the webpack gives to it. Maybe cypress-io/instrument-cra#75 will change its behavior, let me see if I can upgrade that module.

PS: love your project, we definitely need to add it as an example to this repo's README

[penx]

PS: love your project, we definitely need to add it as an example to this repo's README

thanks! I'm working on a blog post to accompany it, but don't want to publish until I can get this thing working. I can send you a draft if you're interested

[penx]

I found this which is loosely related istanbuljs/babel-plugin-istanbul#105

Perhaps in task.js:resetCoverage() you can add something that adds all files to the coverage report as soon as it is initialised (with zero coverage) independently of create-react-app/webpack?

[bahmutov]

@penx can you try v3.5.0 - it should insert dummy placeholders for all files from package.json nyc list (if you use all:true option) into .nyc_output/out.json file before generating a report. It would look something like this:

[penx]

Great, that works! The uncovered files are white and don't have stats against them like number of branches and functions. One side effect of this is the lack of tests doesn't affect the overall code coverage metric. This isn't too important for me right now but perhaps something you may want to look at in the future? I could add a new feature request ticket if you like? Thanks for getting this fixed so quickly.

[penx]

for your known issue Can't configure cypress-specific nyc settings outside of package.json - I have closed this in https://github.com/cypress-io/code-coverage/releases/tag/v3.4.0

I was hoping to have a cypress specific file to put in cypress specific nyc settings, as other parts of the application may also use nyc.

e.g.

  "nyc": {
    "report-dir": "coverage/integration",

The report dir is a cypress specific setting.

Perhaps a nyc or code-coverage: { nyc: { section could be added to cypress.json?

[bahmutov]

For now, I would not want to add any more specific options to Cypress, so yes, please open a new issue about empty coverage for non-included files, ok.

[penx]

On a related note, I noticed when I added "reporter": ["text", "lcov"], to nyc in package.json I got the following error:

     CypressError: `cy.task('coverageReport')` failed with the following error:

> Error: ENOENT: no such file or directory, open '/Users/penx/Development/storybook-code-coverage/coverage/integration/coverage-final.json'

I'll take another look later or tomorrow and open a bug report or have a go at fixing it

[bahmutov]

@penx Ha, I just pushed a fix, this was just a silly bug, I forgot existsSync(...) here

const jsonReportFilename = join(folder, 'coverage-final.json')
if (!existsSync) {
  debug('Did not find final coverage file %s', jsonReportFilename)
  return
}