doctest integration for Cabal

[bitemyapp]

New summary by @ezyang.

The goal of this task is to add a new command Setup doctest, which runs the doctest command on all of the modules of each component in a package, in much the same way Setup haddock runs haddock on all the modules of each component in a package.

There are a number of steps to implementing this:

1. You have to add the boilerplate for setting doctest up as a new command. Grepping for occurrences of haddock should give you an idea for what you have to edit; the key files are Cabal/Distribution/Simple/Setup.hs, adding a case to defaultMainHelper in Cabal/Distribution/Simple.hs, and a new file to actually contain the logic. I'd recommend against putting too many flags into DoctestFlags when starting off.

2. You need to add doctest to the program database, so that you can invoke it (and end-users can customize which binary to use, extra options they want to pass, etc.) The relevant logic is in Cabal/Distribution/Simple/Program/Builtin.hs You should create create something like doctestProgram.

3. The overall logic of the driver should be something like this:

 withAllComponentsInBuildOrder pkg_descr lbi $ \component clbi -> do
   componentInitialBuildSteps (flag haddockDistPref) pkg_descr lbi clbi verbosity
   preprocessComponent pkg_descr component lbi clbi False verbosity suffixes
   -- Do the actual invocation of doctest

Between component, lbi and clbi, you will have all the information necessary to invoke doctest. You can look at https://github.com/haskell/cabal/blob/master/Cabal/Distribution/Simple/Haddock.hs for an example of all of the parts being put together.

4. How should doctest actually be invoked? In theory, doctest accepts flag identical to GHC, so you should be able to reuse Cabal's logic for invoking GHC here. In practice, this is not true, see doctest's argument format does not match GHC's sol/doctest#155 so you will need to apply some workarounds. It might be good to get doctest to fix these discrepancies to solve problems. Still, it is probably wise to reuse the bulk of Cabal's existing logic in https://github.com/haskell/cabal/blob/master/Cabal/Distribution/Simple/GHC.hs#L509 . You may also want to consult https://github.com/phadej/cabal-doctest/blob/master/src/Distribution/Extra/Doctest.hs for "flags that doctest is known to work with". I don't necessarily recommend actually using the code from this module.

---

Original description below:

@sol suggested this would make more sense here than the doctest repository.

A cabal doctest or some manner of Cabal auto-magic is needed.

Should do discovery and running of doctests automatically.

It's extremely hard for people to get doctests working with a Cabal project. It took be 3 days, 10-12 hours apiece just to get doctests working for Bloodhound. Now I realize I'm a fucking moron, but I was also getting help in IRC from people like @ekmett who uses doctests for lens and he had no idea what was causing things to break either.

The things that make doctests hard to use fall into three categories:

1. Cabal project/sandbox awareness
2. Custom Setup.hs for doctest discovery and execution
3. Insanely finicky doctest syntax and interactions with Haddock

This notion fixes 1 and 2, but not 3. However, if users have the reasonable belief that 1 and 2 are okay and should be working, they'll spend less time trying to diagnose the wrong things when they've messed up number 3.

I filed this issue long after having gotten Bloodhound's doctest integration working because I ran into somebody I know on IRC struggling for multiple days with this again.

I'm tired of having to tell people to copy-paste what I have in Bloodhound to make it work and people hate having to cargo-cult this anyway. Doctests have caused problems for the NICTA course as well.

[phadej]

What I can see, than the doctest.hsc used in lens, with test-suite doctests could be directly integrated into cabal, as a "virtual component". Maybe we could add doctest-suite section if there's some configuration needed.

[ezyang]

CC'ing @sol here.

We need to know a little more about how doctest's command line runner works before we can work something out here. I think the main question is whether or not doctest should be implemented like a virtual component (as @phadej suggests), or something more like Haddock, which is not really a component but just another mode you can run your project in.

Based on my reading of https://github.com/sol/doctest/blob/master/src/Run.hs it seems that doctests accepts exactly the same arguments that GHC accepts (have you considered making it a frontend plugin? :) https://downloads.haskell.org/~ghc/master/users-guide/extending_ghc.html#frontend-plugins ) with a little bit of extra smarts to handle options that people commonly forget to add.

My key question: does doctest recompile the module under test, or does it assume that the module has already been compiled and installed? Based on the README, it seems it must be the former. In that case, it seems like a virtual component ala @phadej is highly inappropriate, because you're going to need to also pass in all the flags that you would have needed to build the package in the first place.

Here's my question: why can't doctest be implemented as a wrapper around GHC, passed to Cabal via --with-ghc=ghc-doctest, which both (1) compiles the modules as normal, but (2) also loads and then tests the documentation examples. Because doctest needs to compile the module anyway, so it might as well slot into that codepath. Then it will get all of the package databases, etc., that it needs to load up things.