stack test --coverage fails sporadically

[altaic]

If I run stack test --coverage on a freshly cloned repo (algebra-elementary, in my case), I get

Error: The coverage report for algebra-elementary's test-suite "algebra-elementary-test" did not consider any code. One possible cause of this is if your test-suite builds the library code (see stack issue #1008). It may also indicate a bug in stack or the hpc program. Please report this issue if you think your coverage report should have meaningful results.

But, when I run stack build; stack clean; stack test --coverage, I get

Generating coverage report for algebra-elementary's test-suite "algebra-elementary-test"
 83% expressions used (436/521)
 60% boolean coverage (3/5)
      33% guards (1/3), 1 always True, 1 always False
     100% 'if' conditions (2/2)
     100% qualifiers (0/0)
 81% alternatives used (74/91)
 81% local declarations used (9/11)
 60% top-level declarations used (30/50)

Now, the above is on my laptop, but doing the same stack build; stack clean; stack test --coverage on Travis-CI results in

Error: The coverage report for algebra-elementary's test-suite "algebra-elementary-test" did not consider any code. One possible cause of this is if your test-suite builds the library code (see stack issue #1008). It may also indicate a bug in stack or the hpc program. Please report this issue if you think your coverage report should have meaningful results.

I'm not sure if this is a stack bug, or if my cabal file is messed up. If it's the latter, it may be a good idea to describe the correct cabal configuration in the stack guide, perhaps with a barebones example. I've fiddled with the cabal file quite a bit and haven't figured out how to consistently produce reports (#1008 hasn't really enlightened me).

[rubik]

I, on the other hand, always get the error. For one reason or another, *.tix files are never produced and HPC cannot report anything. This happens even if I add -fhpc to ghc-opts.

[mgsloan]

This is due to HPC using package keys in the tix / mix files, along with a bug in Cabal which causes package keys to vary: haskell/cabal#2870 . The package key is needed in order to filter the report results down to just the library modules (see #634). Unfortunately, the use of package keys rather than IPIDs makes it difficult to munge the tix files to get around this issue.

@rubik Do your test-suites also build the library code (shares hs-source-dirs with the library)? .tix files should be present in the subdirs of $(stack path --local-install-root)/hpc.

A potential workaround for the Cabal issue is to build all the package components together, but this won't be possible when your test-suite depends on some package which depends on the library (cyclic dependency): #1166

Considering that it is reasonably common for packages to define test-suites in this way, and this feature is causing problems, I'm thinking it ought to be turned on with a flag. Perhaps --coverage-only-library.

[rubik]

@mgsloan Thanks, I somehow thought that the .tix files would be in dist/ instead of install/. The files are indeed there, but the HTML index shows no data, while running hpc report results in the following error:

$ stack exec hpc report $(stack path --local-install-root)/hpc/moodle-0.1.0.0/moodle-tests.tix                                          
hpc: can not find Moodle.Types in ["./.hpc"]

I don't understand what I'm doing wrong. Here is my cabal file:
https://github.com/rubik/moodle-to-latex/blob/master/moodle.cabal

[mgsloan]

Yeah, hpc's error reporting isn't so good (see my ghc trac issue on the subject: http://ghc.haskell.org/trac/ghc/ticket/10951 ). Either you have old hpc files there which have a hash mismatch, or the folder isn't there at all.

The mix files are in $(stack path --dist-dir)/hpc/.hpc. So this means the invocation is:

stack exec hpc report $(stack path --local-install-root)/hpc/moodle-0.1.0.0/moodle-tests.tix --hpcdir $(stack path --dist-dir)/hpc/.hpc --reset-hpcdirs

You can see the invocation stack uses by using -v. There will likely be a stack hpc report command in the nearish future. Also, having hpc/.hpc is no longer necessary, and is generally weird. I'm going to change that to just hpc.

[rubik]

Thank you for the quick reply. Unfortunately it still does not work. I tried everything from explicitly adding the -fhpc GHC option to removing my library from the test-suite build-depends... I'll wait for #1191 and stack hpc report.

[mgsloan]

@altaic I've pushed some commits which seem to make things well behaved. In particular, your repro now works fine! As such, I'm closing this issue.

@rubik Feel free to open a new issue if you're still running into problems.