Get coverage per test

[ffMathy]

Using coverlet.core, is it possible to get a "parent line" for a specific hit, to determine the call stack of the covered line?

I want to use that to figure out which tests ran through a specific line.

[ffMathy]

I tried going through the code, but it's a bit too hard to figure out how to debug it properly. Is there a way to debug the tests in the project somehow?

[ffMathy]

If it's not possible to figure out which tests ran through a specific line, could it perhaps be possible to generate a coverage file per test in the test run?

[petli]

I've been curious about getting per-test coverage reports since the file formats tend to support it, but it would require a fair bit of redesign of the hit collection and a challenge to do it efficiently. Right now there is a single int[] per module that the hits are recorded into.

My rough idea of what's necessary to do to create this is that you have to change how methods are instrumented so that on entry they locate the unit test (if any). This would be done by traversing the callstack up, and then cache the result in an AsyncLocal so it doesn't have to be done by every nested instrumented method. A hit array is created for each unit test, and used in the instrumented code instead of the static array that's currently injected.

When the test is done all these hit arrays would have to be transferred over to the coverlet process. This is already a time critical operation that is only stable when using the vstest collector driver, so moving potentially thousands of hit arrays would only be feasible in that driver, excluding the standalone and msbuild drivers as it stands now.

Finally the coverlet core will need to collate these hit arrays to get per-test statistics.

[petli]

If it's not possible to figure out which tests ran through a specific line, could it perhaps be possible to generate a coverage file per test in the test run?

You could do this by scripting your test runs. dotnet test -t lists the tests in a project and iterate over them in a script, and then your script can run test by test by running dotnet test --filter "$testname" --other-parameters-as-needed-to-get-coverage. This will not be very efficient, of course.

[ffMathy]

@petli if I modified CoverletCoverageCollector to do what it does right now for every session, to doing it after every run instead, would that have a huge performance penalty?

And would that work for per-test coverage technically?

[petli]

I don't know much about the collector architecture, but it's a good point to investigate that. May not need any of the call stack analysis or hit array per test, if the array can be fetched and reset between each test instead. That requires that tests are not run in parallel, but that should be a reasonable restriction.

[ffMathy]

Yeah, that's a reasonable restriction for me.

I'm building a tool that can run only affected tests in dotnet, from looking at your GIT changed files.

[ffMathy]

Is it possible to reset between each test right now? Or is that something I should add?

I'd love to contribute on this.

[MarcoRossignoli]

@ffMathy can you elaborate better your scenario?

Collectors could a way and I agree with @petli on the complexity related to redesign current implementation to support this, also we should try to avoid inprocess collectors if possible to avoid the needs of too much config for injection(not so user friendly with runsettings), at the moment in process collectors is embedded inside vstest platform and we're planning to move out to cleanup code the code base.

Another possibility could be expose engine standalone to inject and control hits manually to get an "in memory" report for every test using standard test features i.e. for xunit IAsyncLifetime/Dispose and at the end of every test generate/save something.
In past I did some test on standalone usage here #628 that was asked for a fuzz testing #212 (comment)

[ffMathy]

I want to build a CLI that internally runs tests for you serially (non-parallel) that have been affected by your GIT changes, based on previous coverage.

That's why I want to be able to see what tests ran through a specific line of code.

See the examples given in my other issue here: dotnet/aspnetcore#25194

[ffMathy]

@MarcoRossignoli what do you think about what @petli said here:

May not need any of the call stack analysis or hit array per test, if the array can be fetched and reset between each test instead. That requires that tests are not run in parallel, but that should be a reasonable restriction.

Can you point me in the right direction of implementing such reset functionality, or perhaps how to do it?

As I mentioned earlier, I am running all tests serially one at a time, so there is no parallelism.

[MarcoRossignoli]

If I understood well your tool idea is:

1. Run my tests and create a baseline
2. Update code
3. Run only tests where I updated code reading the mapping 'test name' -> lines covered by that test in last run(from/to ranges)

If you want to use coverlet this could be implemented with collectors, specifically with in-process one because it provides TestStart/TestEnd event.

The workflow of coverlet coverage is:

1. Prepare module: it scans dlls and instruments(add IL hit record method) and "prepare" also a structure where we map hits(array where the array index is related to one source line, in this location we save the number of hits, it's an int[])->lines/branches and depends on drivers you're using we need to preserve this structure to do the correct accounting at the end of coverage.
   For msbuild integration we save this file in a tmp folder.
   For .NET Tool we keep it in memory(it's a console app)
   For collectors we keep this data in memory inside out-of-process collectors https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/VSTestIntegration.md#how-it-works

2. At the end of coverage we "load"(from file or use in memory one) the mapping hits number(position inside array) -> lines/branches and we do accounting. In that way we know that location 23 inside array of hits of x.dll is the line 46 of mySource.cs and was hit n number of times.

The problem now it that you should access at the end of every test to every instrumented dlls like https://github.com/coverlet-coverage/coverlet/blob/master/src/coverlet.collector/InProcDataCollection/CoverletInProcDataCollector.cs#L54 and save/persist current hits array(where there is current test hits) somewhere with the test name and reset hits array(https://github.com/coverlet-coverage/coverlet/blob/master/src/coverlet.core/Instrumentation/ModuleTrackerTemplate.cs#L22).

When all tests ends the control come back to out of process collector https://github.com/coverlet-coverage/coverlet/blob/master/src/coverlet.collector/DataCollection/CoverletCoverageCollector.cs#L158 where you have to create n number of reports for every test.
To do that you need to override your saved hits(n files for every test) with the one used for current report building. Because keep in mind that hits file name is embedded inside the instrumented dll so at the end you'll have n hits array(you saved those after every test end) but only 1 file name on disk where you have to "persist" all hits(one for every dll instrumented and used by test) in the correct place(hits file name https://github.com/coverlet-coverage/coverlet/blob/master/src/coverlet.core/Instrumentation/ModuleTrackerTemplate.cs#L21).

After that your tool could do a diff comparing test/lines with git changes.

As you can see this is no a simple change, also it's very specific and limited to what you needs and the issue with parallel execution is not negligible for big project(today we have already some issue with big repo related to the invasive nature for instance coverage in repo like Roslyn is very very very slow).

I mean the idea of generalize/expose internal coverage engine could help because adding some features(for instance call a callback for every hit, have access to prepared structure in memory etc...) and with brand new collectors used only for your needs(run selected test, totally unrelated to "coverage" problem, you're using coverage only to have a way to know who touch what lines) you could evolve with no "blockers" or dependence of coverlet engine(related only to code coverage and report generation). If we expose core standalone we have only to create the correct contracts(interfaces) but when will be stable won't need great updates(implementation is hidden).

BTW your idea is very cute! If you want discuss further we can setup a call and I can explain better coverlet architecture at the moment there is no documentation and it's a bit complex explain in a post like this and also answer quickly.
We are always looking for help here!

[MarcoRossignoli]

cc: @tonerdo

[ffMathy]

@MarcoRossignoli thank you for that elaborate explanation! However, I still do have some questions.

When would be a good time for a call? I think that's a great idea!

My timezone is UTC+1 though.

Also, how do we make the call? Skype? Discord? Whereby? Zoom?

[ffMathy]

I found a way to get it working I think - I'll have a PR ready soon.

[MarcoRossignoli]

DM on twitter we'll arrange a call on skype https://twitter.com/MarcoRossignoli

[tobyash86]

Is there any progress on this? Are there any plans to put it on roadmap?
Will it support parallel execution or is it fine to implement it with sequential execution?

[MarcoRossignoli]

There's no plan to support this one for the moment, the volume of work to support this one is not trivial.

[tobyash86]

What if I would like to contribute, to let's say implement it in a sequential way? Could you assist with code review and pull request approval when it's done?

[github-actions]

This issue is stale because it has been open for 3 months with no activity.