Skip to content

plz cover inconsistent file count/inclusion across formats and whitespace counted in coverage calculation #3475

@dstaub-work

Description

@dstaub-work

Coverage output formats report inconsistent file counts (XML excludes 0% coverage files)

Description

I have been running plz cover to gather and report code coverage, but I've noticed some inconsistencies with the code coverage data depending on the format that is requested. When running plz cover, the three output formats (stdout, JSON, XML) report different file counts. JSON appears to be the most complete while XML excludes files with 0% coverage. Also, it seems like whitespace might be counted toward coverage figures right now based on some limited investigation.

Steps to Reproduce

Note that I'm filtering to just my Python files, but this occurs for Go files in my repo as well whether I filter to Go or run them all together. I don't normally run a command with all 3 coverage formats, but I've done so here for the sake of reproducing the behavior more simply.

You should be able to reproduce this if you have a mix of files where some have no code coverage from tests.

# Run coverage with all output formats
plz cover --include=py \
    --coverage_results_file=plz-out/test/coverage/py.json \
    --coverage_xml_report=plz-out/test/coverage/py.xml \
    2>&1 | tee /tmp/py_stdout.txt

# Count files in each format
echo "stdout: $(grep -E '\.py [0-9]+/[0-9]+ lines' /tmp/py_stdout.txt | wc -l)"
echo "JSON: $(python3 -c "import json; print(len(json.load(open('plz-out/test/coverage/py.json')).get('files', {})))")"
echo "XML: $(grep -oP 'filename=\"\K[^\"]+' plz-out/test/coverage/py.xml | sort -u | wc -l)"

Results:

Format File Count
stdout 92
JSON 94
XML 83

Discrepancies:

Comparison Missing Files Pattern
JSON → XML 11 All 0% coverage
JSON → stdout 2 1 file at 0%, 1 file at 33% (one uncovered line is whitespace)

Expected behavior:
All three output formats should report the same files. JSON appears to be the most complete and should be the source of truth. Files with 0% coverage are particularly important to include as they represent untested code.

Please Version:
17.27.0 (was previously on 17.14.0, but had the same issue)

Please let me know if I have missed something or this is addressed somewhere! I took a look around and didn't see anything which seemed to be related.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions