Filters not working correctly with 4.6.434-rc

[stephentoub]

We're upgrading https://github.com/dotnet/corefx from 4.6.401-rc to 4.6.434-rc to fix a crash occurring while getting coverage on some of our libraries, and although it fixes the crash, it appears to break a filter. We use a filter -filter:"+[*]* -[*.Tests]*" to exclude our test libraries, but the test assemblies appear to still be showing up in the output (they never did previous to this upgrade)... ReportGenerator then includes them in the report, but with 0% coverage.

[sawilde]

Our dogfood tests run a very similar filter and it's working fine.

Can you point me to your scripts where you run opencover?

[stephentoub]

Here's an example:

RunTestsForProject:
  c:\Users\stoub\Source\repos\corefx\packages/OpenCover\4.6.434-rc\tools\OpenCover.Console.exe -filter:"+[*]* -[*.Tests]*" -excludebyfile:"*\Common\src\System\SR.*" -nodefaultfilters -excludebyattribute:*.ExcludeFromCodeCoverage* -skipautoprops -hideskipped:All -threshold:1 -returntargetcode -register:user -target:c:\Users\stoub\Source\repos\corefx\bin/tests/AnyOS.AnyCPU.Debug\System.Threading.Tasks.Extensions.Tests\dnxcore50/CoreRun.exe -output:c:\Users\stoub\Source\repos\corefx\bin/tests/coverage\System.Threading.Tasks.Extensions.Tests.coverage.xml -targetargs:"xunit.console.netcore.exe System.Threading.Tasks.Extensions.Tests.dll -xml testResults.xml -notrait Benchmark=true  -notrait category=OuterLoop -notrait category=failing "
  ?Executing: c:\Users\stoub\Source\repos\corefx\bin\tests\AnyOS.AnyCPU.Debug\System.Threading.Tasks.Extensions.Tests\dnxcore50\CoreRun.exe
  xUnit.net console test runner (64-bit .NET Core)
  Copyright (C) 2014 Outercurve Foundation.

  Discovering: System.Threading.Tasks.Extensions.Tests
  Discovered:  System.Threading.Tasks.Extensions.Tests
  Starting:    System.Threading.Tasks.Extensions.Tests
  Finished:    System.Threading.Tasks.Extensions.Tests

  === TEST EXECUTION SUMMARY ===
     System.Threading.Tasks.Extensions.Tests  Total: 30, Errors: 0, Failed: 0, Skipped: 0, Time: 0.411s
  Committing...
  Visited Classes 4 of 21 (19.05)
  Visited Methods 30 of 74 (40.54)
  Visited Points 97 of 336 (28.87)
  Visited Branches 34 of 80 (42.5)

  ==== Alternative Results (includes all methods including those without corresponding source) ====
  Alternative Visited Classes 4 of 21 (19.05)
  Alternative Visited Methods 30 of 104 (28.85)
  Touching "c:\Users\stoub\Source\repos\corefx\bin/tests/AnyOS.AnyCPU.Debug\System.Threading.Tasks.Extensions.Tests\dnxcore50\tests.passed.without.OuterLoop.failing".
GenerateIndividualCoverageReport:
  c:\Users\stoub\Source\repos\corefx\packages/ReportGenerator\2.4.0-beta2\tools\ReportGenerator.exe -targetdir:c:\Users\stoub\Source\repos\corefx\bin/tests/cov
  erage\ -reporttypes:Html;Badges -reports:c:\Users\stoub\Source\repos\corefx\bin/tests/coverage\System.Threading.Tasks.Extensions.Tests.coverage.xml
  Loading report 'c:\Users\stoub\Source\repos\corefx\bin\tests\coverage\System.Threading.Tasks.Extensions.Tests.coverage.xml'
   Preprocessing report
   Initiating parser for OpenCover
    Current Assembly: System.Threading.Tasks.Extensions
    Current Assembly: System.Threading.Tasks.Extensions.Tests
  Initializing report builders for report types: Html, Badges
  Analyzing 4 classes
   Creating report 1/4 (Assembly: System.Threading.Tasks.Extensions, Class: System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1)
   Creating report 2/4 (Assembly: System.Threading.Tasks.Extensions, Class: System.Runtime.CompilerServices.ValueTaskAwaiter`1)
   Creating report 3/4 (Assembly: System.Threading.Tasks.Extensions, Class: System.Threading.Tasks.ValueTask`1)
   Creating report 4/4 (Assembly: System.Threading.Tasks.Extensions.Tests, Class: System.Threading.Tasks.Channels.Tests.ValueTaskTests)
   Creating summary
  Report generation took 0.4 seconds

You can see that even though we have that filter in places, it's still generating data for System.Threading.Tasks.Extensions.Tests, which should be filtered out.

If you want to try this yourself, you can do:

git clone https://github.com/dotnet/corefx.git
build.cmd /p:Coverage=true

Unfortunately the first time you run build.cmd it's going to take a while, probably half an hour, as it downloads a whole bunch of packages and builds the whole repo. After that, you can test it much more easily by doing:

cd src\System.Threading.Tasks.Extensions\tests
msbuild /t:rebuildandtest /p:Coverage=true

That'll go into just one of the library projects and build and run just its tests with code coverage. This library is small and it should take just a few seconds.

[stephentoub]

Also, you can see the difference just by upgrading from 401-rc to 434-rc. First I use 434-rc, then all I do is change the "434-rc" in the command to "401-rc"... check out the difference in the number of classes processed:

c:\Users\stoub\Source\repos\corefx\bin\tests\AnyOS.AnyCPU.Debug\System.Threading.Tasks.Extensions.Tests\dnxcore50>c:\Users\stoub\Source\repos\corefx\packages/OpenCover\4.6.434-rc\tools\OpenCover.Console.exe -filter:"+[*]* -[*.Tests]*" -excludebyfile:"*\Common\src\System\SR.*" -nodefaultfilters -excludebyattribute:*.ExcludeFromCodeCoverage* -skipautoprops -hideskipped:All -threshold:1 -returntargetcode -register:user -target:c:\Users\stoub\Source\repos\corefx\bin/tests/AnyOS.AnyCPU.Debug\System.Threading.Tasks.Extensions.Tests\dnxcore50/CoreRun.exe -output:c:\Users\stoub\Source\repos\corefx\bin/tests/coverage\System.Threading.Tasks.Extensions.Tests.coverage.xml -targetargs:"xunit.console.netcore.exe System.Threading.Tasks.Extensions.Tests.dll -xml testResults.xml -notrait Benchmark=true  -notrait category=OuterLoop -notrait category=failing "
Executing: c:\Users\stoub\Source\repos\corefx\bin\tests\AnyOS.AnyCPU.Debug\System.Threading.Tasks.Extensions.Tests\dnxcore50\CoreRun.exe
xUnit.net console test runner (64-bit .NET Core)
Copyright (C) 2014 Outercurve Foundation.

Discovering: System.Threading.Tasks.Extensions.Tests
Discovered:  System.Threading.Tasks.Extensions.Tests
Starting:    System.Threading.Tasks.Extensions.Tests
Finished:    System.Threading.Tasks.Extensions.Tests

=== TEST EXECUTION SUMMARY ===
   System.Threading.Tasks.Extensions.Tests  Total: 30, Errors: 0, Failed: 0, Skipped: 0, Time: 0.402s
Committing...
Visited Classes 4 of 21 (19.05)
Visited Methods 30 of 74 (40.54)
Visited Points 97 of 336 (28.87)
Visited Branches 34 of 80 (42.5)

==== Alternative Results (includes all methods including those without corresponding source) ====
Alternative Visited Classes 4 of 21 (19.05)
Alternative Visited Methods 30 of 104 (28.85)

c:\Users\stoub\Source\repos\corefx\bin\tests\AnyOS.AnyCPU.Debug\System.Threading.Tasks.Extensions.Tests\dnxcore50>c:\Users\stoub\Source\repos\corefx\packages/OpenCover\4.6.401-rc\tools\OpenCover.Console.exe -filter:"+[*]* -[*.Tests]*" -excludebyfile:"*\Common\src\System\SR.*" -nodefaultfilters -excludebyattribute:*.ExcludeFromCodeCoverage* -skipautoprops -hideskipped:All -threshold:1 -returntargetcode -register:user -target:c:\Users\stoub\Source\repos\corefx\bin/tests/AnyOS.AnyCPU.Debug\System.Threading.Tasks.Extensions.Tests\dnxcore50/CoreRun.exe -output:c:\Users\stoub\Source\repos\corefx\bin/tests/coverage\System.Threading.Tasks.Extensions.Tests.coverage.xml -targetargs:"xunit.console.netcore.exe System.Threading.Tasks.Extensions.Tests.dll -xml testResults.xml -notrait Benchmark=true  -notrait category=OuterLoop -notrait category=failing "
Executing: c:\Users\stoub\Source\repos\corefx\bin\tests\AnyOS.AnyCPU.Debug\System.Threading.Tasks.Extensions.Tests\dnxcore50\CoreRun.exe
xUnit.net console test runner (64-bit .NET Core)
Copyright (C) 2014 Outercurve Foundation.

Discovering: System.Threading.Tasks.Extensions.Tests
Discovered:  System.Threading.Tasks.Extensions.Tests
Starting:    System.Threading.Tasks.Extensions.Tests
Finished:    System.Threading.Tasks.Extensions.Tests

=== TEST EXECUTION SUMMARY ===
   System.Threading.Tasks.Extensions.Tests  Total: 30, Errors: 0, Failed: 0, Skipped: 0, Time: 0.536s
Committing...
Visited Classes 4 of 4 (100)
Visited Methods 30 of 30 (100)
Visited Points 97 of 97 (100)
Visited Branches 34 of 34 (100)

==== Alternative Results (includes all methods including those without corresponding source) ====
Alternative Visited Classes 4 of 4 (100)
Alternative Visited Methods 30 of 30 (100)

c:\Users\stoub\Source\repos\corefx\bin\tests\AnyOS.AnyCPU.Debug\System.Threading.Tasks.Extensions.Tests\dnxcore50>

[sawilde]

@stephentoub please try https://www.nuget.org/packages/OpenCover/4.6.476-rc

[ddur]

We know that reverting change will work. I think your fix was good enough.
Maybe we should make process and assembly filters case insensitive as they are in windows file systems?

[sawilde]

The assemblyName does not from from the file system it comes from a call to GetAssemblyInfo and is not a path nor is it a filename - because as you have seen the filename and assemblyName do not always match.

[ddur]

Only one exception comes from mscorlib system, not available to mortals. In that case only assembly name is provided to filters. Other cases can be case sensitivity, we can deal with it.

Without path we can't deal with cases where two identical assembly names loaded into memory as in #364

[sawilde]

Without path we can't deal with cases where two identical assembly names loaded into memory as in #364

rather than make filters any more complex perhaps we should introduce an exclude assemblies in folder e.g. -excludepath or -excludepaths option instead. And perhaps a repeatable sample to add to the unit/bdd tests to stop regression

[ddur]

With path, we can exclude everything from system with single filter *:\WINDOWS\*
Current assembly filters will never match path because of .ext(ension) and leading \(backslash)
Except *name*, but that will match both path and name, I think is OK.