[Bug]: Code coverage metric is misleading — generated interop code dominates the denominator

[azchohfi]

Describe the bug

The code coverage produced by the test run (dotnet run --project WinApp.Cli.Tests ... --coverage --coverage-output-format cobertura, as invoked by scripts/build-cli.ps1) reports a wildly misleading overall number.

Measured on main:

• Reported overall line-rate: ~18% (lines-covered 18,358 / lines-valid 101,587).
• The denominator (101,587 lines) is dominated by auto-generated interop code emitted into obj\**, none of which is meaningful to cover:
  • Generated\CsWin32\Windows.Win32.NativeMethods.g.cs (thousands of lines of P/Invoke thunks)
  • Microsoft.Interop.ComInterfaceGenerator\... COM interface shims (Direct3D11 ID3D11DeviceContext/ID3D11Device, UIAutomation IUIAutomation*, IImageList, IMoniker, IAccessible, …)
  • System.Text.RegularExpressions.Generator\...RegexGenerator.g.cs
  • Both the Debug and Release obj\ trees are counted, double-counting the generated files.

When the generated obj\** code is excluded, the real, hand-written source measures:

• ~48.8% line coverage — 176 files, 21,127 valid lines, 10,306 covered.

So the headline metric under-reports true coverage by ~30 points and can't be used as a quality gate or trend signal. It also silently double-counts generated files across build configurations.

Note on hardware/COM/GPU code: ~2,300 lines of the hand-written source (the real UiAutomationService, WgcCapture screen capture, and the SendInput mouse/keyboard wrappers) are currently exercised only by a separate PowerShell E2E harness, not the MSTest unit suite, so they show as uncovered. These are not to be excluded from the denominator — they are real product code and will be brought under test (unit-testable logic seams plus an in-process UI test that drives the real service against the WinUI sample app). The only legitimate exclusion is the generated code above.

To Reproduce

Steps to reproduce the behavior:

1. From a clean checkout, build the test project: dotnet build src/winapp-CLI/WinApp.Cli.Tests/WinApp.Cli.Tests.csproj -c Release
2. Run the suite with coverage exactly as scripts/build-cli.ps1 does: dotnet run --project src/winapp-CLI/WinApp.Cli.Tests/WinApp.Cli.Tests.csproj -c Release --no-build --coverage --coverage-output-format cobertura
3. Open the produced *.cobertura.xml and read the root line-rate / lines-valid / lines-covered attributes.
4. Observe line-rate ≈ 0.18 with lines-valid ≈ 101,587; inspect the per-<class> filename attributes and see the largest entries are files under obj\...\Generated\ and ...Generator\.

Expected behavior

• Coverage should be measured against hand-written product source only. Generated files (obj\**, *.g.cs, CsWin32 / ComInterfaceGenerator / RegexGenerator output) must be excluded from both numerator and denominator via a checked-in coverage configuration (e.g. a .runsettings Sources/ModulePaths exclude, consumed by the --coverage run and build-cli.ps1).
• No hand-written service or logic is excluded. The hardware/COM/GPU interop that the unit suite currently fakes (real UiAutomationService, WgcCapture, keyboard/mouse SendInput) stays in the denominator and is covered by real tests: unit tests for its logic seams, plus an in-process UI test that launches the WinUI sample app and drives the genuine service (inspect/search/invoke/set-value/wait-for/screenshot + real type/click) so --coverage instruments those paths.
• The reported overall number should reflect real coverage and be stable across Debug/Release (no double-counting), so it can be used as a trend signal / gate.

Screenshots

N/A

OS Version and details

Windows 11 / Windows Server (net10.0-windows10.0.19041.0). Not OS-specific.

---

Update — second cause: the Release build itself under-counts line coverage

Excluding generated code (above) is necessary but not sufficient. The --coverage run in build-cli.ps1 also collected coverage on an optimized Release build. On Release, the C# compiler often drops or merges the sequence points for standalone block-brace lines ({/}) and duplicate return statements, so the Microsoft coverage engine reports many brace-only lines as hits=0 even when the enclosing code fully executes. This systematically deflates line coverage and caps control-flow-heavy files at ~70–80% regardless of test quality.

Measured with the same 1,637 tests, generated code already excluded:

	Overall (hand-written)	ManifestService.cs
Release (understated)	~48.8%	62–71%
Debug (accurate)	~59.5%	81.7%

ManifestService.cs reads 81.7% in Debug on main (no new tests) — already above the ~73% "ceiling" it appeared to hit in Release, proving the ceiling was a measurement artifact, not un-coverable code.

The fix is to collect coverage on a Debug build (the standard configuration for .NET coverage), which maps every line faithfully — no brace-filtering heuristic needed. Fixed in #637: coverage-report.ps1 defaults to -Configuration Debug, and build-cli.ps1 runs the test/coverage pass in Debug (passing -p:TreatWarningsAsErrors=true to preserve the Release-only warning gate). The shipped CLI is still the Release dotnet publish.

The honest baseline is therefore ~59.5%, not the ~48.8% first reported here.