Unexpected Coverage Changes

[halibobo1205]

What is code coverage?

Code coverage is a measurement used to express which lines of code were executed by a test suite. We use three primary terms to describe each line executed.

• hit indicates that the source code was executed by the test suite.
• partial indicates that the source code was not fully executed by the test suite; there are remaining branches that were not executed.
• miss indicates that the source code was not executed by the test suite.

Coverage is the ratio of hits / (sum of hit + partial + miss). A code base that has 5 lines executed by tests out of 12 total lines will receive a coverage ratio of 41% (rounding down).

Phrased simply, code coverage provides a visual measurement of what source code is being executed by a test suite. This information indicates to the software developer where they should write new tests in an effort to achieve higher coverage.

Testing source code helps to prevent bugs and syntax errors by executing each line with a known variable and cross-checking it with an expected output.

Percentages on the commit page

These percentages are near identical to what's on a PR, but the comparison for change is a little different. In a pull request you compare a head to a base, but in a commit, the change is determined by comparing to the previous commit on your branch. These percentages are also specific to the commit itself, not the wider PR/MR that the commit is a part and allow you to see a more granular view of how your coverage is changing based on the work done in that single commit.

Introduction

There are many reasons why coverage may change in unexpected ways. Codecov analyzes the pull/commit diff, detecting coverage changes on both lines of code that changed and lines that were not changed.

Indirect coverage changes are differences of coverage to lines of code that are not adjusted in the pull/commit diff. This can occur for a number of reasons. In the UI we call these changes "indirect changes" - on the pull request page you will see an "indirect changes" tab and navigating there will bring you a list of all of the files impacted by indirect changes. More often than not an unexpected coverage change is an indirect change.

Reasons for unexpected changes

There are a number of reasons that cause line coverage to change unexpectedly. These can be due to:

1. Adding or removing tests.
2. Failing to upload coverage reports, or a different number of reports between head and base
3. Time-sensitive tests (e.g., one case before 2pm UTC, a different case after 2pm UTC).
4. Missing coverage reports or failed builds.
5. Dependencies changed resulting in a different execute plan.
6. Encrypted variables may prevent some execution paths.
7. In the case of changes to a PR, the base or head commits may not have had an uploaded report, or may not be the commit you expect
8. Different error handling paths: sometimes exception handlers are tested and some are not, oftentimes the root of "flaky" tests
9. Due to long running test builds or dynamic test selection, you are intentionally not running all tests on every commit.

👍 Indirect Coverage Changes
Almost every example above will manifest as an indirect coverage change in the "indirect changes" portion of the pull request page shown in the screenshot below.

If you want to know more, please refer to https://docs.codecov.com/docs/about-code-coverage

Expected behaviour

Less unexpected coverage changes as possible, CI works in normal.

Actual behaviour

CI often fails due to unexpected coverage changes.

[halibobo1205]

Time-sensitive tests

[halibobo1205]

Different error handling paths

[halibobo1205]

conditional judgment or config switch