Wrong coverage when using isolate: false

[RodrigoFerretti]

Describe the bug

This is the same bug as mentioned in #3846, but now with minimal reproduction.

I've read somewhere that vitest orders the execution by the file's size, descending.

The coverage is ignored for the smaller tests when the method being tested is called in one of the "setupFiles" configured, even though everything should be covered

Reproduction

https://github.com/RodrigoFerretti/vitest-coverage-w-isolate

System Info

System:
    OS: Linux 5.15 Ubuntu 22.04.4 LTS 22.04.4 LTS (Jammy Jellyfish)
    CPU: (8) x64 Intel(R) Core(TM) i7-10610U CPU @ 1.80GHz
    Memory: 5.28 GB / 15.23 GB
    Container: Yes
    Shell: 5.8.1 - /usr/bin/zsh
  Binaries:
    Node: 18.19.0 - ~/.nvm/versions/node/v18.19.0/bin/node
    Yarn: 1.22.22 - ~/.yarn/bin/yarn
    npm: 10.2.3 - ~/.nvm/versions/node/v18.19.0/bin/npm
  Browsers:
    Chrome: 127.0.6533.72
  npmPackages:
    @vitest/coverage-v8: 2.0.3 => 2.0.3 
    vite: 5.3.4 => 5.3.4 
    vitest: 2.0.3 => 2.0.3

Used Package Manager

yarn

Validations

• Follow our Code of Conduct
• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
• The provided reproduction is a minimal reproducible example of the bug.

[AriPerkkio]

Does this happen when setup files are not importing source files?

[RodrigoFerretti]

@AriPerkkio Removing the "multiply" function from the "setup.ts" file fixes the problem for this reproduction code, but it is impossible for me to do something like this in the project I am working on. I kept the isolate true option to keep the coverage, even though the tests take longer like this.

[AriPerkkio]

Thanks for checking. It helps us debugging when we can narrow down the required conditions to "setupFiles + isolate: false" only.

[RodrigoFerretti]

Hi! Any updates on this? Is there any way I can help? I really would like to use isolate:false to speed up the test run.

[AriPerkkio]

On @vitest/coverage-istanbul the root cause is here. We shouldn't remove the files from coverage object, instead reset all counters to 0.

vitest/packages/coverage-istanbul/src/index.ts

Lines 18 to 20 in 8878b04

	// Reset coverage map to prevent duplicate results if this is called twice in row
	// @ts-expect-error -- untyped global
	globalThis[COVERAGE_STORE_KEY] = {}

On @vitest/coverage-v8 there's something weird going on. I spent couple of hours debugging it but didn't yet find root cause or could reproduce the issue outside Vitest.

[AriPerkkio]

And @vitest/coverage-v8 works fine with isolate: false, as long as setupFiles are not imporing the source files. You could try some work-arounds there in your project, if possible.

Need to check if we are running setupFiles outside vite-node's node:vm, directly on NodeJS. That could confuse V8's profiler like this, as the filename remains same but executed content is different. 🤔