Code coverage shows different code coverage result for JS and HTML tests

[motss]

Description

This issue was recently found when both JS and HTML tests are involved in a test suite. wtr enables us to write tests in either JS and/ or HTML. However, the code coverage tends to tell a different story when both JS and HTML tests present in a test suite when some of the tests have to be written in HTML while the rest can be written in JS files.

Please clone this repo to better understand the issue.

Expected behavior

Code coverage should show 100% when all lines of code are covered by tests.

Actual behavior

Code coverage drops below 100% when all lines of code are covered by JS tests but partially in HTML tests.

Code coverage output

Code coverage with JS tests only

All codes are covered in JS tests and it shows 100% code coverage.

Code coverage with JS and HTML tests

All codes are covered in JS tests but not in HTML test. This causes the code coverage to drop below 100% even though all lines of code are covered in JS tests.

[LarsDenBakker]

Do you see any useful clues in the generated coverage report? We rely on chromium to do this instrumentation. You could also use babel for generating coverage: https://modern-web.dev/docs/test-runner/writing-tests/code-coverage/#coverage-browser-support

[motss]

I'm just guessing if this line would be the cause. It seems to me that the coverage from the same file path will be replaced by another coverage collected from different test file. So, the coverage from the HTML test might replace the coverage from the JS test. Not sure if that would be case, just wild guessing here.

[LarsDenBakker]

That merges the test coverage of the all the test files.

I'm not sure I fully understand the case you're describing. We do measure the code coverage of the JS code in the HTML file. So when you look at the coverage report, you should see what is and isn't covered.

[motss]

It's been days since my last message. I finally got the time to figure out how the coverageMaps are collected in wtr.

What I'm suspecting is

web/packages/test-runner-core/src/coverage/getTestCoverage.ts

Line 101 in cd8928b

addingMissingCoverageBranches(coverages);

might be doing something weird that causes such behavior which is related to istanbuljs/istanbuljs#531.

From the screenshot below, it seems to me that some coverage branches are being added wrongly for some reason which might be due to v8 coverage:

The branch number is 4 in JS + HTML tests. 1 extra branch has been added for the same file for unknown reason

The branch number is 3 in JS-only tests