Python Coverage does not work

[ulfjack]

I thought that there was an existing issue for this, but I can't find it.

I was able to successfully collect coverage for python code with a modified version of coverage.py:

1. Use a Bazel binary that include b01c859
2. Check out the modified coverage.py from https://github.com/ulfjack/coveragepy/tree/lcov-support (this is based on the patch at DO-NOT-MERGE: Patched-on LCOV file format support for coveragepy v4.x nedbat/coveragepy#863 with a small change to make it work with Bazel)
3. Run bazel:

bazel coverage --test_env=PYTHON_COVERAGE=/path/to/coveragepy/coverage/__main__.py //python:lib_test

Test coverage should be written to bazel-testlogs/python/lib_test/coverage.dat in lcov format.

[ulfjack]

In order for cross-language coverage to work in Bazel, all coverage data needs to be converted into a common format. The above workaround uses an upstream patch to coverage.py. As an alternative, we could also change LcovMerger to support coverage.py's output format.

Reference to upstream's issue: nedbat/coveragepy#587

[dmadisetti]

The issue you were looking for may have been under python_rules bazel-contrib/rules_python#43

Thanks for the suggested workaround!

[zuerst]

@ulfjack thank you for the suggestion. Unfortunately I am seeing empty coverage.dat file with some warnings in in the test.log file. Any idea what I might be missing?

test.log

Jul 08, 2020 6:35:19 PM com.google.devtools.coverageoutputgenerator.Main getTracefiles
INFO: No lcov file found.
Jul 08, 2020 6:35:19 PM com.google.devtools.coverageoutputgenerator.Main getGcovInfoFiles
INFO: No gcov info file found.
Jul 08, 2020 6:35:19 PM com.google.devtools.coverageoutputgenerator.Main getProfdataFileOrNull
INFO: No .profdata file found.
Jul 08, 2020 6:35:19 PM com.google.devtools.coverageoutputgenerator.Main main
WARNING: There was no coverage found.

Also is there a documentation about using Bazel Coverage on python that might be helpful to read?

[ulfjack]

The most likely reason for coverage not to work is an incorrect --instrumentation_filter, and the second most likely is having sources in a *_test rule - test rules do not collect any coverage. Beyond that, there are a number of things that could go wrong, but I don't know which one it is. You'd have to post a repro if you need more help.

[chickenandpork]

@ulfjack I get empty coverage files regardless what I try, and these are repositories with only __init__.py and test_...py files in the srcs attribute. The code under test in one project are in libraries and referenced as dependencies (ie not in srcs, but also not an embed of any kind).

Do you have a basic example that you used during development? Or a test case that works? It would be really helpful to see any sort of functioning test to fast-track some experimentation.

For reference, I've tried various combinations of this script, changing the parameters of the last line to get a run that doesn't fail, but gives non-zero coverage.dat files. So far, never ever a nonzero coverage.dat.

#!bash

COVERAGEPY=${HOME}/src/coveragepy
test -d "${COVERAGEPY}" || git clone   -b lcov-support    https://github.com/ulfjack/coveragepy.git "${COVERAGEPY}"

# According to https://github.com/bazelbuild/bazel/issues/10660, and some assumption:

bazel coverage --test_env=PYTHON_COVERAGE=${COVERAGEPY}/coverage/__main__.py  //:test_ns

# Now, in other documentation, there is much discussion about local_jdk and various coverage flags.  
# The following completes without error, but still empty coverage.dat files
#
# NOTE: //:test_ns is the only test target for this example.

bazel coverage -s  --test_env=PYTHON_COVERAGE=${COVERAGEPY}/coverage/__main__.py  --sandbox_debug \
    --instrument_test_targets   --instrumentation_filter=//...:all --combined_report=lcov \
    --javabase=@bazel_tools//tools/jdk:remote_jdk11  //:test_ns

Sometimes I also try with --coverage_report_generator= @bazel_tools//tools/test/CoverageOutputGenerator/java/com/google/devtools/coverageoutputgenerator:Main but there seems no difference.

[dmadisetti]

@chickenandpork I've been generating reports for a while now. Not directly with bazel, but by manually calling grcov in conjunction with coveragepy. Here's a sample from one of my private repo actions:

    - name: Python coverage
      run: |
        cd "${GITHUB_WORKSPACE}/src"
        curl -L https://github.com/ulfjack/coveragepy/archive/lcov-support.tar.gz | tar xvz
        ~/bin/bazel coverage -t- --instrument_test_targets --experimental_cc_coverage \
            --test_output=errors --linkopt=--coverage --linkopt=-lc \
            --test_env=PYTHON_COVERAGE=${GITHUB_WORKSPACE}/src/coveragepy-lcov-support/__main__.py \
            --define=config_file=test //python/:tests
        # Unfortunately, coverage does not produce the hit information. Running
        # the compiled tests explicitly seems to work though.
        find bazel-bin/python/ -maxdepth 1 -regex ".*-test$" -exec {} \;
        cp bazel-out/k8-fastbuild/testlogs/pd3/python/smoke-test/coverage.dat coverage.py.info
      env:
        CC: clang-9

    - name: codecov
      if: always()
      run: |
        set -x
        cd "${GITHUB_WORKSPACE}/src"
        ~/bin/grcov coverage.py.info bazel-bin/python/_objs/ -t lcov \
          --ignore "external/*" --ignore "/usr/*" \
          --ignore "*deps_*" --ignore "*_pb2.py"  \
          --llvm > lcov.py.info
        sed -i 's/SF:.*test\.runfiles\/python\/python/SF:python/g' coverage.py.info
        cd ..
        bash -x <(curl -s https://codecov.io/bash) -Z -v -f src/lcov.py.info -F python_tests
      env:
        CC: clang-9
        CODECOV_TOKEN: ${{ secrets.CODE_COV }}

[lberki]

/cc @oquenchil

Maybe this can fit into your coverage sprint?