-
Notifications
You must be signed in to change notification settings - Fork 212
Description
Coverage output formats report inconsistent file counts (XML excludes 0% coverage files)
Description
I have been running plz cover to gather and report code coverage, but I've noticed some inconsistencies with the code coverage data depending on the format that is requested. When running plz cover, the three output formats (stdout, JSON, XML) report different file counts. JSON appears to be the most complete while XML excludes files with 0% coverage. Also, it seems like whitespace might be counted toward coverage figures right now based on some limited investigation.
Steps to Reproduce
Note that I'm filtering to just my Python files, but this occurs for Go files in my repo as well whether I filter to Go or run them all together. I don't normally run a command with all 3 coverage formats, but I've done so here for the sake of reproducing the behavior more simply.
You should be able to reproduce this if you have a mix of files where some have no code coverage from tests.
# Run coverage with all output formats
plz cover --include=py \
--coverage_results_file=plz-out/test/coverage/py.json \
--coverage_xml_report=plz-out/test/coverage/py.xml \
2>&1 | tee /tmp/py_stdout.txt
# Count files in each format
echo "stdout: $(grep -E '\.py [0-9]+/[0-9]+ lines' /tmp/py_stdout.txt | wc -l)"
echo "JSON: $(python3 -c "import json; print(len(json.load(open('plz-out/test/coverage/py.json')).get('files', {})))")"
echo "XML: $(grep -oP 'filename=\"\K[^\"]+' plz-out/test/coverage/py.xml | sort -u | wc -l)"
Results:
| Format | File Count |
|---|---|
| stdout | 92 |
| JSON | 94 |
| XML | 83 |
Discrepancies:
| Comparison | Missing Files | Pattern |
|---|---|---|
| JSON → XML | 11 | All 0% coverage |
| JSON → stdout | 2 | 1 file at 0%, 1 file at 33% (one uncovered line is whitespace) |
Expected behavior:
All three output formats should report the same files. JSON appears to be the most complete and should be the source of truth. Files with 0% coverage are particularly important to include as they represent untested code.
Please Version:
17.27.0 (was previously on 17.14.0, but had the same issue)
Please let me know if I have missed something or this is addressed somewhere! I took a look around and didn't see anything which seemed to be related.