cabal and integration tests or how to get code coverage for binaries

[sophie-h]

My project structure:

• library
• binary (depends library)
• unit tests
• integration tests (written in pytest)

The integration tests call the binary. To get code coverage for those calls, I tried to call pytest via system from an extra cabal test-suite. Turns out that this doesen't help. Also wrapping the binary into the test-suite (details) does not help. Seems like only the call from cabal test counts into the coverage report.

So I went back to calling the binary directly and generating reports via hpc manually. First problem: I need to add an obscure --hpcdir for the library (dist/hpc/dyn/mix/<bin>-<version>/<random stuff>/). Second: I get 100% (0/0) coverage. So the library is not included in the report of the binary?

This was all working when I didn't had the split into library/binary. Is it really impossible to build a binary+library with working coverage via cabal?

[ezyang]

See also #3682. They don't look the same but similar issues show up in the discussion.

[sophie-h]

Okay, I got the manual hpc working again.

Why did I get 100% (0/0) coverage? Turns out if I supply a random extra (positional) argument to hpc report, it always gives this output. So important: Supplying two hpcdirs only works via writing --hpcdir two times.

Since all hpc errors are suppressed with this extra argument, my path's above are wrong. I only need to supply

hpc report my.tix \
  --hpcdir dist/hpc/vanilla/mix/<name> \
  --hpcdir dist/hpc/vanilla/mix/<name>-0.1.0.0

Still, that I need the current version number does not really simplify automation.

I think this can only be simplified if cabal would have explicit support for integration tests?

[ezyang]

The Cabal library for coverage tests really does need some love. I haven't fully investigated your report, but it definitely sounds plausible; would you be interested in helping contribute some patches to make this do the right thing? (We'll give you commit access!)

[sophie-h]

would you be interested in helping contribute some patches to make this do the right thing?

Totally. But that's more like, I can try. Don't know if I can match up to the cabal code base.

Probably, this need some conceptual work first. Maybe one could say that I am talking about integration tests, while cabal only supports unit tests. Right now I think it would be good to introduce another TestType.

I think there is little input that we can get from other build systems since they have even less features. I will try to investigate a bit nevertheless.

[ezyang]

@qua-bla Yeah. To start off, what would a hypothetical manual section explaining how to do coverage look like? The current support is very specialized: http://cabal.readthedocs.io/en/latest/installing-packages.html?highlight=coverage#cmdoption-setup-configure--enable-coverage ; basically it is something like, "If you pass --enable-coverage, if you run a test suite, the locally defined library will have coverage run for it." (There is even fancy footwork to get the test suite to NOT show up in the coverage report.) What would a more flexible interface look like?

I think new-build can help a bit here; for example, if you are profiling, you may want to control the profiling detail on various packages. You can do that by placing configuration inside a package section for each appropriate one. So maybe there should be some sort of "coverage flag" which indicates whether or not a package's library should be included in code coverage metrics"? Perhaps there is just some engineering work to be done so that coverage can be generated for executables (I don't think another test type is the right thing to do.)

[atodorov]

@sophie-h can you provide some more details about how you got coverage working for binaries? I have the same setup as you do:

1. library + unit tests
2. several binaries + python based integration tests

when using --enable-coverage and then running the binaries they produce .tix files but I have no idea how to amend the report from cabal with that info.

@ezyang I can give a try improving the current state of affairs, starting with docs (#4401). Can you give me some hints, places in the code to look at, briefly explain how the internal cabal+hpc machinery works, etc ?

[ezyang]

@atodorov Sorry about the late reply, we would definitely be happy for contributions.

There are two primary components to hpc in Cabal (library). First is building the program with hpc support. This is handled in Distribution/Simple/GHC.hs. I'd recommend reading the GHC manual on code coverage https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/profiling.html#observing-code-coverage which will familiarize you with what command line options are available.

Second is actually running the program and generating the coverage report. As coverage is currently only supported in test and bench, it is not surprising that this code is invoked from Distribution/Simple/Test/ExeV10.hs and implemented in Distribution/Simple/Hpc.hs. markupTest and markupPackage are particularly key players. Also grep for Coverage.

Let me know if you have other questions.