Program coverage crashes in cabal new-test

[jiaochangyang]

Preprocessing test suite 'testsuite' for simple-server-0.1.0.0..
Building test suite 'testsuite' for simple-server-0.1.0.0..
Running 1 test suites...
Test suite testsuite: RUNNING...
Tests
  core unit tests
    testing storeInsert empty:                                     OK
    testing storeInsert when already in small:                     OK
    testing storeInsert when already in commit:                    OK
    testing storeInsert when data is in both:                      OK
    testing isSecondaryIndexed when in store:                      OK
    testing isSecondaryIndexed when not in store:                  OK
    testing secondaryLookup when not in secondary:                 OK
    testing secondaryLookup when in small:                         OK
    testing secondaryLookup when in commit:                        OK
    testing secondaryInsert when not in secondary:                 OK
    testing read (ReadAll) when not in store:                      OK
    testing read (lookup) when in small store:                     OK
    testing read (lookup) when in commit store:                    OK
    testing write successfully in small store:                     OK
    testing write unsuccessfully (duplicate) in small store:       OK
    testing write unsuccessfully (duplicate) in commit store:      OK
    testing write failure in store (somehow):                      OK
    testing write failure in secondary (already in store somehow): OK
    testing write failure in store (already in secondary somehow): OK

All 19 tests passed (0.00s)
Test suite testsuite: PASS
Test suite logged to:
/Users/cyj/Documents/simple-server/dist-newstyle/build/x86_64-osx/ghc-8.2.1/simple-server-0.1.0.0/c/testsuite/test/simple-server-0.1.0.0-testsuite.log
hpc: can not find simple-server-0.1.0.0-inplace/Data.API.Read in ./.hpc, /Users/cyj/Documents/simple-server/dist-newstyle/build/x86_64-osx/ghc-8.2.1/simple-server-0.1.0.0/c/testsuite/hpc/vanilla/mix/simple-server-0.1.0.0, /Users/cyj/Documents/simple-server/dist-newstyle/build/x86_64-osx/ghc-8.2.1/simple-server-0.1.0.0/c/testsuite/hpc/vanilla/mix/testsuite
CallStack (from HasCallStack):
  error, called at libraries/hpc/Trace/Hpc/Mix.hs:122:15 in hpc-0.6.0.3:Trace.Hpc.Mix
cabal: Tests failed for test:testsuite from simple-server-0.1.0.0.

@cartazio was looking through this with me and can reproduce the same issue

we are using ghc 8.2.1 and cabal new-build / new-test

cabal --version
cabal-install version 2.0.0.0
compiled using version 2.0.0.2 of the Cabal library

a verbose (-v3) output would be as follows

te testsuite: PASS
Test suite logged to:
/Users/cyj/Documents/simple-server/dist-newstyle/build/x86_64-osx/ghc-8.2.1/simple-server-0.1.0.0/c/testsuite/test/simple-server-0.1.0.0-testsuite.log
Environment: [("Apple_PubSub_Socket_Render","/private/tmp/com.apple.launchd.Q3JKpyPRfN/Render"),("COLORFGBG","7;0"),("COLORTERM","truecolor"),("HASKELL_DIST_DIR","/Users/cyj/Documents/simple-server/dist-newstyle/build/x86_64-osx/ghc-8.2.1/simple-server-0.1.0.0/c/testsuite"),("HOME","/Users/cyj"),("ITERM_PROFILE","Default"),("ITERM_SESSION_ID","w0t2p0:F80CC278-087E-4137-B59D-639A0978E05D"),("LANG","en_US.UTF-8"),("LOGNAME","cyj"),("PATH","/Users/cyj/.ghc-installs/ghc-8.2.1-hq/bin:/Users/cyj/.cabal/bin:/Users/cyj/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/cyj/.cabal/bin:/Users/cyj/.cabal/bin"),("PWD","/Users/cyj/Documents/simple-server"),("SECURITYSESSIONID","186a6"),("SHELL","/bin/bash"),("SHLVL","1"),("SSH_AUTH_SOCK","/private/tmp/com.apple.launchd.AaBcIdWfTT/Listeners"),("TERM","xterm-256color"),("TERM_PROGRAM","iTerm.app"),("TERM_PROGRAM_VERSION","3.1.2"),("TERM_SESSION_ID","w0t2p0:F80CC278-087E-4137-B59D-639A0978E05D"),("TMPDIR","/var/folders/fp/_6rb74hd3j39xp_scrxxf4ww0000gn/T/"),("USER","cyj"),("XPC_FLAGS","0x0"),("XPC_SERVICE_NAME","0"),("__CF_USER_TEXT_ENCODING","0x1F5:0x0:0x0")]
/Users/cyj/.ghc-installs/ghc-8.2.1-hq/bin/hpc markup /Users/cyj/Documents/simple-server/dist-newstyle/build/x86_64-osx/ghc-8.2.1/simple-server-0.1.0.0/c/testsuite/hpc/vanilla/tix/testsuite/testsuite.tix '--destdir=/Users/cyj/Documents/simple-server/dist-newstyle/build/x86_64-osx/ghc-8.2.1/simple-server-0.1.0.0/c/testsuite/hpc/vanilla/html/testsuite' '--hpcdir=/Users/cyj/Documents/simple-server/dist-newstyle/build/x86_64-osx/ghc-8.2.1/simple-server-0.1.0.0/c/testsuite/hpc/vanilla/mix/testsuite' '--hpcdir=/Users/cyj/Documents/simple-server/dist-newstyle/build/x86_64-osx/ghc-8.2.1/simple-server-0.1.0.0/c/testsuite/hpc/vanilla/mix/simple-server-0.1.0.0' '--exclude=Main'
hpc: can not find simple-server-0.1.0.0-inplace/Data.API.Read in ./.hpc, /Users/cyj/Documents/simple-server/dist-newstyle/build/x86_64-osx/ghc-8.2.1/simple-server-0.1.0.0/c/testsuite/hpc/vanilla/mix/simple-server-0.1.0.0, /Users/cyj/Documents/simple-server/dist-newstyle/build/x86_64-osx/ghc-8.2.1/simple-server-0.1.0.0/c/testsuite/hpc/vanilla/mix/testsuite
CallStack (from HasCallStack):
  error, called at libraries/hpc/Trace/Hpc/Mix.hs:122:15 in hpc-0.6.0.3:Trace.Hpc.Mix
/Users/cyj/.ghc-installs/ghc-8.2.1-hq/bin/hpc returned ExitFailure 1
packages definitely up to date: simple-server-0.1.0.0-inplace
packages previously probably up to date: simple-server-0.1.0.0-inplace, simple-server-0.1.0.0-inplace-simple-server-scotty, simple-server-0.1.0.0-inplace-simple-server-snap
packages now probably up to date: simple-server-0.1.0.0-inplace, simple-server-0.1.0.0-inplace-simple-server-scotty, simple-server-0.1.0.0-inplace-simple-server-snap
packages newly up to date:
packages out to date: simple-server-0.1.0.0-inplace-testsuite
packages invalid due to dep change: simple-server-0.1.0.0-inplace-testsuite
packages invalid due to build failure: simple-server-0.1.0.0-inplace-testsuite
CallStack (from HasCallStack):
  die', called at ./Distribution/Client/ProjectOrchestration.hs:782:8 in main:Distribution.Client.ProjectOrchestration
cabal: Tests failed for simple-server-0.1.0.0-inplace-testsuite.

we can make the simple application code available to any cabal maintainer privately (well, @cartazio will deal with that), but can't make it public at the moment, (its a pretty small project, but minimizing this failure would require more ideas about how cabal works than we do! :) )

[cartazio]

pinging @ttuegel and @hvr and @bgamari and @23Skidoo :)

i want my program coverage magic!

it does seem to correctly generate mix and tix files ... i think ... but i never have tried coverage checking plus new build before :(

[23Skidoo]

@ttuegel is the one most knowledgeable about these bits of Cabal.

[cartazio]

I’ll try to dig into it / help however I can to facilitate... I’m just not familiar with this stuff :)

…

On Wed, Sep 27, 2017 at 4:25 PM Mikhail Glushenkov ***@***.***> wrote: @ttuegel <https://github.com/ttuegel> is the one most knowledgeable about these bits of Cabal. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4798 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAQwuubmcpKOkk9K_XgpoUXonnXE6qwks5smq8_gaJpZM4PmOv6> .

[cartazio]

@23Skidoo @ttuegel lemme know if theres any way i can help
... should i use cabal with stack trace, or dig into the two source locations with those errors?

[EButlerIV]

@cartazio
I just replicated it myself, and I don't think it's project- or local settings-dependent. This is what happened when I ran it against a freshly downloaded package from hackage.

The test suite log doesn't contain any errors, but my build dir definitely doesn't contain a .hpc directory. It's likely that any coverage files generated are being stored somewhere else.

eugene@Eugenes-MacBook-Pro ~/D/P/blah> cabal get u2f
Downloading u2f-0.1.0.1...
Unpacking to u2f-0.1.0.1/
eugene@Eugenes-MacBook-Pro ~/D/P/blah> cd u2f-0.1.0.1/
eugene@Eugenes-MacBook-Pro ~/D/P/b/u2f-0.1.0.1> cabal new-test --enable-coverage
[snip]
Finished in 0.0223 seconds
5 examples, 0 failures
Test suite hspec-suite: PASS
Test suite logged to:
/Users/eugene/Development/Personal/blah/u2f-0.1.0.1/dist-newstyle/build/x86_64-osx/ghc-8.0.2/u2f-0.1.0.1/c/hspec-suite/test/u2f-0.1.0.1-hspec-suite.log
hpc: can not find u2f-0.1.0.1-inplace/U2F.Types in ./.hpc, /Users/eugene/Development/Personal/blah/u2f-0.1.0.1/dist-newstyle/build/x86_64-osx/ghc-8.0.2/u2f-0.1.0.1/c/hspec-suite/hpc/vanilla/mix/u2f-0.1.0.1, /Users/eugene/Development/Personal/blah/u2f-0.1.0.1/dist-newstyle/build/x86_64-osx/ghc-8.0.2/u2f-0.1.0.1/c/hspec-suite/hpc/vanilla/mix/hspec-suite
CallStack (from HasCallStack):
  error, called at libraries/hpc/Trace/Hpc/Mix.hs:122:15 in hpc-0.6.0.3:Trace.Hpc.Mix
cabal: Tests failed for test:hspec-suite from u2f-0.1.0.1.

E: Manually moving the files to the expected location results in successful generation of coverage reports. It looks like this is the issue.:

eugene@Eugenes-MacBook-Pro ~/D/P/b/u2f-0.1.0.1> mkdir .hpc
eugene@Eugenes-MacBook-Pro ~/D/P/b/u2f-0.1.0.1>
cp -r dist-newstyle/build/x86_64-osx/ghc-8.0.2/u2f-0.1.0.1/hpc/vanilla/mix/u2f-0.1.0.1/u2f-0.1.0.1-inplace .hpc/

eugene@Eugenes-MacBook-Pro ~/D/P/b/u2f-0.1.0.1> ls -lh .hpc/
total 0
drwxr-xr-x  4 eugene  staff   136B Sep 30 15:45 u2f-0.1.0.1-inplace
eugene@Eugenes-MacBook-Pro ~/D/P/b/u2f-0.1.0.1> cabal new-test --enable-coverage
In order, the following will be built (use -v for more details):
 - u2f-0.1.0.1 (test:hspec-suite) (first run)
Preprocessing test suite 'hspec-suite' for u2f-0.1.0.1..
Building test suite 'hspec-suite' for u2f-0.1.0.1..
Running 1 test suites...
Test suite hspec-suite: RUNNING...

Request/Response Parsing
  should work for registration requests
Registration Flow
  should check that request and response challenges are equivalent
  should validate properly formed request and response
Signin Flow
  should check that request and response challenges are equivalent
  should validate properly formed request and response

Finished in 0.0056 seconds
5 examples, 0 failures
Test suite hspec-suite: PASS
Test suite logged to:
/Users/eugene/Development/Personal/blah/u2f-0.1.0.1/dist-newstyle/build/x86_64-osx/ghc-8.0.2/u2f-0.1.0.1/c/hspec-suite/test/u2f-0.1.0.1-hspec-suite.log
Writing: u2f-0.1.0.1-inplace/U2F.Types.hs.html
Writing: u2f-0.1.0.1-inplace/U2F.hs.html
Writing: hpc_index.html
Writing: hpc_index_fun.html
Writing: hpc_index_alt.html
Writing: hpc_index_exp.html
Test coverage report written to
/Users/eugene/Development/Personal/blah/u2f-0.1.0.1/dist-newstyle/build/x86_64-osx/ghc-8.0.2/u2f-0.1.0.1/c/hspec-suite/hpc/vanilla/html/hspec-suite/hpc_index.html
1 of 1 test suites (1 of 1 test cases) passed.
Writing: u2f-0.1.0.1-inplace/U2F.Types.hs.html
Writing: u2f-0.1.0.1-inplace/U2F.hs.html
Writing: hpc_index.html
Writing: hpc_index_fun.html
Writing: hpc_index_alt.html
Writing: hpc_index_exp.html
Package coverage report written to
/Users/eugene/Development/Personal/blah/u2f-0.1.0.1/dist-newstyle/build/x86_64-osx/ghc-8.0.2/u2f-0.1.0.1/c/hspec-suite/hpc/vanilla/html/u2f-0.1.0.1/hpc_index.html

[EButlerIV]

Ok, last update. It looks like the real issue is caused by bad arguments in this use of hpc markup. The last hpcdir argument appears to be incorrect.

This is what it uses

/Users/eugene/Development/Personal/blah/u2f-0.1.0.1/dist-newstyle/build/x86_64-osx/ghc-8.0.2/u2f-0.1.0.1/c/hspec-suite/hpc/prof/mix/u2f-0.1.0.1

This is what it should actually be

/Users/eugene/Development/Personal/blah/u2f-0.1.0.1/dist-newstyle/build/x86_64-osx/ghc-8.0.2/u2f-0.1.0.1/hpc/prof/mix/

Incorrect invocation in full

/Users/eugene/.installed-ghc/ghc-8.0.2/bin/hpc markup /Users/eugene/Development/Personal/blah/u2f-0.1.0.1/dist-newstyle/build/x86_64-osx/ghc-8.0.2/u2f-0.1.0.1/c/hspec-suite/hpc/prof/tix/hspec-suite/hspec-suite.tix --destdir=/Users/eugene/Development/Personal/blah/u2f-0.1.0.1/dist-newstyle/build/x86_64-osx/ghc-8.0.2/u2f-0.1.0.1/c/hspec-suite/hpc/prof/html/hspec-suite --hpcdir=/Users/eugene/Development/Personal/blah/u2f-0.1.0.1/dist-newstyle/build/x86_64-osx/ghc-8.0.2/u2f-0.1.0.1/c/hspec-suite/hpc/prof/mix/hspec-suite --hpcdir=/Users/eugene/Development/Personal/blah/u2f-0.1.0.1/dist-newstyle/build/x86_64-osx/ghc-8.0.2/u2f-0.1.0.1/hpc/prof/mix/u2f-0.1.0.1 --exclude=Main

E: Can confirm that normal cabal test does run and create coverage reports without issue when coverage is enabled.

[cartazio]

so either
a) ghc isn't being told where to write the mix files correctly by cabal
b) cabal isn't telling hpc where to read the mix files correctly ?

[cartazio]

it seems that cabal new-test didn't adjust how args/paths are passed to the the markup routines that invoke hpc... though @EButlerIV and i are really just poking around and guessing atm :)

that is: when we build ghc programs with coverage, /Users/eugene/Development/Personal/blah/u2f-0.1.0.1/dist-newstyle/build/x86_64-osx/ghc-8.0.2/u2f-0.1.0.1/c/hspec-suite/hpc/prof/mix/u2f-0.1.0.1 should be setup properly ... or something

[cartazio]

@23Skidoo would fixing this be also a bug fix for 2.0 series cabal :)

[cartazio]

a) possible culprit: program coverage flags not set correctly for new build in general?
shouldn't these two definitions of hpc dir

/Users/carter/Desktop/repoScratcher/CabalSept2017/Cabal/Cabal/Distribution/Simple/GHC.hs:
  543        -- TH always needs default libs, even when building for profiling
  544  
  545:   -- Determine if program coverage should be enabled and if so, what
  546    -- '-hpcdir' should be.
  547:   let isCoverageEnabled = libCoverage lbi
  548        -- TODO: Historically HPC files have been put into a directory which
  549        -- has the package name.  I'm going to avoid changing this for
  ...
  554        hpcdir way
  555          | forRepl = mempty  -- HPC is not supported in ghci
  556:         | isCoverageEnabled = toFlag $ Hpc.mixDir distPref way pkg_name
  557          | otherwise = mempty
  558  
  ...
 1121    -- modules?  FIX: what about exeName.hi-boot?
 1122  
 1123:   -- Determine if program coverage should be enabled and if so, what
 1124    -- '-hpcdir' should be.
 1125:   let isCoverageEnabled = exeCoverage lbi
 1126        distPref = fromFlag $ configDistPref $ configFlags lbi
 1127        hpcdir way
 1128          | gbuildIsRepl bm   = mempty  -- HPC is not supported in ghci
 1129:         | isCoverageEnabled = toFlag $ Hpc.mixDir distPref way (gbuildName bm)
 1130          | otherwise         = mempty
 1131  

is the results of distBuildDirectory not being passed correctly somewhere?

[23Skidoo]

would fixing this be also a bug fix for 2.0 series cabal :)

It depends on how hard it will be to backport.

[ttuegel]

The root of this issue is that the wrong build directory is being passed to the test phase. The test phase is getting dist-newstyle/build/x86_64-linux/ghc-8.0.2/u2f-0.1.0.1/t/hspec-suite when it should be getting dist-newstyle/build/x86_64-linux/ghc-8.0.2/u2f-0.1.0.1. I have no idea why or where this happens because I still cannot wrap my head around the new-* phases.

[ttuegel]

One more thing: even if I can fix that, there is the underlying problem that each component now has a different build directory, which is incompatible with the existing API of hpcDir. In other words, I could not backport this hypothetical bug fix to Cabal 2.0.

[cartazio]

@ttuegel if myself chang and Eugene can help please layout how we can! A fix for master would probably still be a great improvement. And we could help figuring out a backport once we see the diff. In 2.0 they’re all in the same component afaict so it should be a simpler version of the same fix I think

…

On Thu, Oct 12, 2017 at 9:35 AM Thomas Tuegel ***@***.***> wrote: One more thing: even if I can fix that, there is the underlying problem that each component now has a different build directory, which is incompatible with the existing API of hpcDir. In other words, I could not backport this hypothetical bug fix to Cabal 2.0. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4798 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAQwjFqvHt7UHGWj568ZG1sDs--BE0Eks5srhWvgaJpZM4PmOv6> .

[ezyang]

Oh, I guess the problem here makes sense: for a non-Custom setup, new-test will create separate build directories for the library and test (like, literally separate, with each having their own LocalBuildInfo). And I guess the coverage mechanism was never clued in about how to deal with this. So this is more of a feature request.

One way to work around the problem, then, is to turn off per-component build. I think there are some knobs for this, but you can at least easily check by changing the build-type to Custom.