Code coverage with "all" option does not report uncovered source code

[toaditoad]

Link to bug demonstration repository

https://github.com/toaditoad/code-coverage-mwe

Expected Behavior

Code coverage should be less than 100% because there is only one test spec FooPage.spec.js covering the source code of FooPage.vue. BarPage.vue should be reported as uncovered.

Observed Behavior

The code coverage is 100%. The all option of nyc includes all source code but does not indicate untouched source code (like BarPage.vue) as uncovered.

Troubleshooting steps

• still occurring when I put cache: false in my nyc config

Environment Information

$ npx envinfo@latest --preset nyc
npx: installed 1 in 0.773s

  System:
    OS: Linux 4.15 Ubuntu 18.04.5 LTS (Bionic Beaver)
    CPU: (8) x64 Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
    Memory: 15.50 GB / 31.27 GB
  Binaries:
    Node: 14.16.1 - /usr/local/bin/node
    Yarn: 1.22.5 - /usr/bin/yarn
    npm: 6.14.12 - /usr/local/bin/npm
  npmPackages:
    @babel/register: 7.14.5 => 7.14.5 
    @istanbuljs/nyc-config-babel: 3.0.0 => 3.0.0 
    @vue/cli-plugin-babel: 4.5.13 => 4.5.13 
    babel-core: 7.0.0-bridge.0 => 7.0.0-bridge.0 
    babel-loader: 8.2.2 => 8.2.2 
    babel-plugin-istanbul: 6.0.0 => 6.0.0 
    babel-preset-typescript-vue: 1.1.1 => 1.1.1 
    source-map-support: 0.5.19 => 0.5.19 
    typescript: 4.2.2 => 4.2.2 

Possibly related issues

• nyc --all option reporting wrong coverage results if all files are instrumented #1372
• How to use istanbul's --include-all-sources flag? babel-plugin-istanbul#197
• Coverage for untested files? babel-plugin-istanbul#105

see also https://stackoverflow.com/questions/68194900/cypress-code-coverage-with-nyc-all-option-does-not-report-source-code-as-unco

[toaditoad]

The problem seems to be caused by cypress-io/code-coverage#207 and its related PR cypress-io/code-coverage#208 .

See Readme of my updated working example for a (not so nice) workaround to add all source files as additional entries in Cypress' webpack preprocessor: https://github.com/toaditoad/code-coverage-mwe

[craigsumner]

I'm glad to see I'm not the only one confused and dismayed by this behavior.
Terrible code coverage comes out looking not-so-bad because of this issue.
I would love to see what our actual code coverage is, but I can't get the workaround working - possibly because we're on cypress 7?

[this-miguel]

any new on this? I'm having the same issue trying to get coverage report for tests using cypress, I pre-instrument my code with babel-plugin-istanbul, when trying to generate the report with nyc even if I use the option all set to true, the coverage only shows files that were used when running the cypress tests any file not used on the tests execution does not appear on the coverage reports.

[WallaceChen123]

@toaditoad I also encountered this issue and I believe this is a serious issue of coverage statistics, this causes the code coverage is incorrect. Do you have any workaround to solve this issue for Cypress 12? Thanks in advance.

[toaditoad]

I'm not even sure if we use nyc any more. Having a quick look at our project, I can tell that vue 3.4.22, cypress-vite 1.5.0, vite 5.2.9 and vite-plugin-istanbul 5.0.0, vitest 1.5.0 is used. Unfortunately, I don't have the time to generate a new minimal working example. However, we have never found a workaround for the originally reported issue. Code coverage seems fine now but so much has changed that I cannot pinpoint the decisive factor.