[llvm-cov] Add support for 'llvm-cov show' and 'llvm-cov report' to optionally accept coverage counts from (modified) JSON produced by 'llvm-cov export'

[bartlettroscoe]

Description

It would be very desirable for the commands llvm-cov report and llvm-cov show to optionally accept raw coverage data as JSON (modified after being generated by llvm-cov export) , instead of only from the *.profdata files. The JSON coverage data might be input as the new argument --cov-data <coverage>.json called like:

llvm-cov show --cov-data <coverage>.json [other options as they are now]

and

llvm-cov report --cov-data <coverage>.json [other options as they are now]

and the new argument --cov-data <coverage>.json would take the place of the argument -instr-profile PROFILE and perhaps even the [BIN] [-object BIN] ... binary arguments?

This would allow running different test suites for the same library code, exporting coverage as JSON from llvm-cov export for each test suite (e.g., testsuite1 and testsuite2) and doing things like:

• Adding together the coverage from the two test suites (i.e. taking the union)
• Diffing the coverage between the two test suites (i.e. what coverage is only in testsuite1, only in testsuite2, or in both test suites)

Then you could produce different outputs of coverage files for each of these use cases. For example, you could call llvm-cov show with the input JSON that contains the coverage in testsuite2 that is not provided in testsuite1. In this scenario, testsuite1 might be a fast running unit-test suite while testsuite2 might be a much more expensive set of system-level tests. This would show you where you might be able to add some fast unit tests to provide the missing coverage. Also, this data would be needed for doing things like test suite/case reduction or in replacing one set of tests with another set of tests.

Another important use case is diffing the coverage before and after changes to the code in a GitHub PR or GitLab MR. You could highlight the lines of code that are newly covered and/or lost coverage after the proposed set of Git commits. Also, you might require the PR/MR to only add coverage and not loose any coverage in region or branch coverage and that any regions (or branches) that are touched must be covered with tests. The latter would force the usage of the Legacy Code Change Algorithm for all code in the repo with every change. That way, the coverage will only improve (or stay the same) with every commit to the repo. (And for legacy code, the coverage would necessarily go up as development continues.)

These are incredibly important use cases to make coverage information useful and the basic tools should support these use cases. That is, absolute coverage is not very compelling or motivating for most people and project stakeholders. But targeted differences in coverage like described above can be very compelling and can be enforced by reasonable development and integration processes in an incremental way.

Workarounds

One could diff the generated *.html files produced by llvm-cov show for testsuite1 and testsuite2 (e.g. using a tool like html-differ) and produce new *.html files for the difference or the union. But that would be more expensive and fragile and require more custom user code to implement compared to working with the data in the JSON files produced by llvm-cov export.

[bartlettroscoe]

For that matter, it would be great of llvm-cov could support commands like:

llvm-cov diff --cov-data <testsuite1>.json --cov-data-ref <testsuite2>.json

and

llvm-cov union --cov-data <testsuite1>.json --cov-data-ref <testsuite2>.json

that would robustly diff or take the union of the coverage data in the two files <testsuite1>.json and <testsuite2>.json and return the diffed/unioned JSON data to STDOUT.

And the issues for robustly diffing the coverage data are similar to the issues for taking the union of coverage data when the regions and branches to fully match up.

Then people would not need to be writing their own custom tools for doing this sort of thing.

[bartlettroscoe]

But again, as a workaround, we could likely just work with the generated *.html files produced from llvm-cov show but that is a different type of effort.

However, that approach has the advantage that we could color code the lines based on different criteria. For example, for a GitHub PR/GitLab MR you could color lines/branches as follows:

• Green: Lines or branches that went from uncovered to covered
• Orange: Lines or branches that went from covered to uncovered.

Then developers and reviewers could clearly see the changes in coverage that result from a given PR/MR.

The same type of coloring could be done for comparing any two test suites.

It just occurred to me that llvm-cov show could be directly extended to produce these extra colored lines/branches by taking in the argument --cov-data-ref <reference>.json. Then lines could be colored based on changes in coverage (provided in -instr-profile PROFILE [BIN] [-object BIN] ... or --cov-data <coverage>.json) itself. This would be very useful.

Similarly, llvm-cov report could be extended to show differences in coverage metrics by passing in a reference set of coverage with --cov-data-ref <reference>.json. (Would need to define the extra columns needed for this.)

Having these features would make coverage incredibly useful. (Currently, people mostly just ignore absolute coverage because it is not actionable.)