Fix misleading code coverage measurement (#630) - #632
Merged
Conversation
The `--coverage` run counted auto-generated interop (CsWin32 P/Invoke thunks, ComInterfaceGenerator COM shims, RegexGenerator) emitted into obj\**, inflating the denominator to ~101k lines and reporting a meaningless ~18% overall. The real hand-written surface is ~21k lines at ~48.8%. This changes only the measurement, not test content: - Add src/winapp-CLI/coverage.runsettings excluding generated code only (obj\**, *.g.cs, *.Designer.cs, [GeneratedCode]). No hand-written service or logic is excluded -- the hardware/COM/GPU interop stays in the denominator and will be covered by real tests in follow-up PRs. - Wire --coverage-settings into scripts/build-cli.ps1 so CI and local runs match. - Add scripts/coverage-report.ps1: runs the suite with the settings, parses the Cobertura report over product source only, prints per-directory + top-uncovered breakdown and an overall %, with an optional -Threshold gate. - Document the coverage scope/policy in the tests README. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot started reviewing on behalf of
Alexandre Zollinger Chohfi (azchohfi)
July 15, 2026 04:57
View session
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes misleading code coverage measurement for the winapp CLI by excluding only auto-generated code from the coverage denominator, aligning CI and local reporting with the real hand-written product surface (per #630).
Changes:
- Adds
src/winapp-CLI/coverage.runsettingsto exclude generated sources (obj\**,*.g.cs,*.Designer.cs) and[GeneratedCode]from coverage. - Wires the runsettings into
scripts/build-cli.ps1so CI/local--coverageruns use the same exclusions. - Adds
scripts/coverage-report.ps1to run coverage and produce a product-source-focused report (per-directory + top uncovered), and documents the policy inWinApp.Cli.Tests/README.md.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| src/winapp-CLI/WinApp.Cli.Tests/README.md | Documents the “exclude generated only” coverage policy and how to measure coverage consistently. |
| src/winapp-CLI/coverage.runsettings | Introduces a coverage configuration to exclude generated code from the denominator. |
| scripts/coverage-report.ps1 | Adds a helper script to run coverage with the runsettings and report coverage over hand-written product source. |
| scripts/build-cli.ps1 | Updates the existing test+coverage invocation to apply the checked-in runsettings. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Contributor
Build Metrics ReportBinary Sizes
Test Results✅ 1637 passed, 1 skipped out of 1638 tests in 500.5s (+88.6s vs. baseline) Test Coverage❌ 48% line coverage, 53.5% branch coverage · ✅ +29.9% vs. baseline CLI Startup Time32ms median (x64, Updated 2026-07-15 05:14:46 UTC · commit |
Nikola Metulev (nmetulev)
approved these changes
Jul 15, 2026
Alexandre Zollinger Chohfi (azchohfi)
added a commit
that referenced
this pull request
Jul 15, 2026
…637) Follow-up to #632. That PR fixed the biggest coverage distortion (generated code inflating the denominator), but it still collected coverage on an optimized **Release** build — where the Microsoft coverage engine reports many standalone block-brace lines (`{`/`}`) as `hits=0` even when the code executes (it drops or merges their sequence points). That deflates line coverage and caps control-flow-heavy files at ~70–80%. ## Fix Collect coverage on a **Debug** build, which maps lines faithfully (the standard configuration for .NET coverage). No test content changes. - `coverage-report.ps1`: default `-Configuration Debug` (+ rationale in help). - `build-cli.ps1`: build + run the test/coverage pass in Debug so the CI-posted cobertura `line-rate` matches local runs. Pass `-p:TreatWarningsAsErrors=true` on the Debug build to keep the warning-as-error gate that `Directory.Build.props` otherwise applies only to Release. The shipped CLI is still the Release `dotnet publish` — only the test run moves to Debug, and the Release artifact stays exercised end-to-end by the sample/npm E2E suites. - `README`: document why Debug and correct the baseline figure. ## Impact on the honest baseline | | Overall | ManifestService.cs | |---|---|---| | Release (understated) | ~48.8% | 62–71% | | **Debug (accurate)** | **~59.5%** | **81.7%** | Same tests, ~11 points higher — the gap was pure Release brace under-counting. This makes the number CI posts on PRs honest and unblocks the per-file 95% target for the follow-up coverage PRs (the ~70–80% "ceiling" was a measurement artifact, not real code that couldn't be covered). ## Review Ran the repo's `pr-review` skill (7 specialists + a GPT multi-model cross-check) and validated at runtime: - Full Debug coverage run (1,637 pass / 1 skipped; 3 pre-existing env-only E2E failures) → 59.5%. - Debug build with `-p:TreatWarningsAsErrors=true` → 0 warnings, 0 errors. - Confirmed packaging / MSIX / NuGet / docs-generation consume the Release `publish` output, not the solution `bin`; the CI metrics baseline transitions cleanly. Closes out the Release-measurement half of #630. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #630.
Problem
The MSTest
--coveragerun counted auto-generated interop — CsWin32 P/Invoke thunks (NativeMethods.g.cs),ComInterfaceGeneratorCOM shims (D3D11 / UI Automation), andRegexGeneratorstate machines — emitted intoobj\**. That inflated the denominator to ~101k lines and reported a meaningless ~18% overall. The real hand-written surface is ~21k lines at ~48.8%.This is the first PR of the test-coverage initiative and changes only the measurement, not any test content, so every later coverage PR is measured against an honest denominator.
Changes
src/winapp-CLI/coverage.runsettings(new) — a Microsoft Code Coverage runsettings that excludes generated code only:obj\**,*.g.cs,*.Designer.cs, and the[GeneratedCode]attribute. No hand-written service or logic is excluded — the hardware/COM/GPU interop (UiAutomationService,WgcCapture, keyboard/mouse input) stays in the denominator and will be covered by real tests (unit logic-seams + an in-process UI test) in follow-up PRs.scripts/build-cli.ps1— wires--coverage-settingsinto the existing test run so CI and local numbers match.scripts/coverage-report.ps1(new) — dev/CI tool: runs the suite with the settings, parses the Cobertura report over product source only (de-duping line hits across partial classes / build configs), and prints a per-directory + top-uncovered breakdown and an overall %. Supports-Area,-Filter,-Threshold(gate),-CoberturaPath,-CsvOut. Propagates the test run's exit code so it can't green-light a run whose tests failed.src/winapp-CLI/WinApp.Cli.Tests/README.md— documents the coverage scope and the "generated code is the only exclusion" policy.Coverage scope policy
Baseline (measured with this tooling)
Overall 48.71% — 176 files, 10,286 covered / 21,116 valid lines. Per-directory: Services ~41%, Commands ~63%, Helpers ~59%, others tracked in #630.
Validation
coverage-report.ps1run end-to-end (Release): prints the full breakdown and, on the current suite's pre-existing failures, correctly still emits coverage and exits non-zero (validated: exit code 2 propagated).dist/cli.js-missing E2E tests (runnpm run buildinsrc/winapp-npm), unrelated to this PR.🤖 Reviewed with the winappcli
pr-reviewskill (8 specialists + an independent GPT-family cross-check) before opening; findings addressed.