A full code coverage report allows you to check for each part of the code if there is a test that currently covers it by executing the relevant lines of code. Unfortunately the motivation to increase code coverage using full reports is low, and the effort to understand why no existing mzcompose-based test covers the code in question is high.
Thus to prevent regressions and new bugs we instead want to ensure that new and changed productive code in a pull request (PR) is adequately tested. This makes sense both when code is refactored as well as when new functionality is added. For each PR you or someone from the QA team can use the Coverage Buildkite pipeline to trigger a coverage build and test run with coverage information for it. After the test runs the resulting coverage information will be aggregated and analyzed to indicate which changed lines of productive code are not covered.
As an example let's take the adapter: Update privileges when an owner is updated pull request. To run PR code coverage against it, go to the Coverage Buildkite pipeline, click on "New Build" and enter the full commit hash and branch with your Github username:
After some time your Coverage run will be finished and contain a summary of any uncovered lines in the form of a git diff where each line highlighted with a ! is not covered:
We can see that some of the newly added code is not covered. This is the point where we should think about if we can write an explicit test case that tests this code adequately.
Code that is only covered by unit tests, but not by any end-to-end test, is highlighted with a . instead of !.
Additionally you can also download the full coverage report in HTML format from the artifacts of the "Analyze code coverage for PR" step. This should only be necessary if you are interested in seeing coverage of the entire product:
The Coverage Buildkite pipeline runs a subset of tests from the Test pipeline, which is already automatically run against each pull request. The excluded tests are marked with a coverage: skip in the pipeline.template.yml:
- Miri test: Subset of cargo test, no additional use
- Maelstrom coverage of persist: Randomized testing probably shouldn't count for coverage, not stable
- Feature benchmark (Kafka only): Benchmarks shouldn't count for coverage
materialize/ci/test/pipeline.template.yml
Lines 155 to 162 in 8147c0b
We don't run code coverage automatically against each PR yet. This has three reasons:
- Code coverage runs can take twice as much time as regular test runs
- Timing-related issues could only show up without coverage, so we can't just replace the normal tests with coverage tests
- Running both tests and coverage tests more than doubles our infrastructure cost for each PR
Thus we are at the moment running code coverage manually for the PRs. You are welcome to trigger coverage for your own PRs though.
Unfortunately there are many known issues in Rust's code coverage. These were encountered in our code base:
- Missing coverage for entire function with #[tracing::instrument(...)]
- Bad coverage with macro_rules!
For this reason you can't trust the coverage data 100%. For now we can just use it as guidance to find code we expect to be tested, but which is not.