Add flexibility for .err and .out comparison in cabal-install integration tests

[BardurArantsson]

As discussed in #2868 (comment) it would be nice to have a little more flexibility for the matching used for ".err" and ".out" files in the cabal-install integration tests. This would allow for better handling of Windows (".exe" appears in the "program name" portion of output) and perhaps also in other cases where a little flexibility is required.

(Mostly copied from that thread:)

A reasonable start would be line-by-line regex comparison where each .out or .err file could contain lines like:

   RE:this is a regex
   this is not

If we want to go super-advanced, something like http://llvm.org/docs/CommandGuide/FileCheck.html could also be an option. (I think this may be overkill for our purposes since cabal-install output is usually pretty simple if you can control the inputs.)

[BardurArantsson]

@23Skidoo : Turns out, I can't assign myself. Oh, GitHub Issues how I love thee :).

[23Skidoo]

@BardurArantsson Just gave you and @phadej Write permissions, so self-assigning should work from now on.

[23Skidoo]

One thing I like about FileCheck is that both the test and the expected output are combined in a single file, so tests are more self-contained. In principle, we could do the same thing.

Also, if we want to reuse the same approach in the test suite for Cabal proper, then it probably makes sense to package IntegrationTests.hs as a separate library (say, tasty-cabal-integration-tester).

[BardurArantsson]

I don't think the "integration-test" approach works at all for Cabal itself, really. While doing the integration-tests thing I had a quick look at the PackageTests for Cabal, and it appears that the Cabal PackageTester.hs (and sometimes the Check.hs for each individual test) contains basically a more hardcoded subset of cabal-install which is necessary to run the checks.

I don't think we'll (realistically) ever have fully self-contained tests -- after all we'll always need .cabal files and directories with sources, etc. etc.

A bit off-topic, but seeing the redundancy between Cabal/PackageTests and cabal-install reminds me: Overall, it appears that the Cabal/cabal-install split is a very strange and not entirely helpful one. It seems to be split in a sort of horizontal way:

|   (world)     |
|---------------|
| cabal-install |
|---------------|
|    Cabal      |
|---------------|

whereas I think it's often more helpful to split things in a vertical way, like (hypothetically):

|  (world)     |  (world)   |   (world)   | ...
|--------------+------------+-------------+
|              |            |  commands   | ...
|--------------+------------+-------------+
| file-formats |   solver   |             | ...
|--------------+------------+-------------+

where commands could be sort of like the current cabal-install (perhaps as a library) and would depend on solver, file-formats, and whatever else it needed. The main point here would be to separate along functionality lines rather than some artificial "abstraction level".

Anyway, this should probably be on the mailing list and not here...

[ezyang]

@BardurArantsson The historic motivation for the abstraction barrier is that Cabal looks at one package at a time, cabal-install looks at multiple packages. One might argue that Cabal/cabal-install should be modularized further (the file-formats library), but historically it's been pretty annoying to do development on multiple packages so it's not been done.

[BardurArantsson]

@ezyang Oh, right. That does kind of make sense -- so cabal-install essentially started as a "solver" of sorts and evolved into the beast we have today :).

[BardurArantsson]

Just FTR: It's pretty unlikely that I'll actually get around to this any time in the near future (and the demand is probably quite limited), so if anyone wants to pick it up, be my guest. I'd be happy to help.

[BardurArantsson]

(Added label "easy" to the 'start-of-line-regex-marker' version since it requires very little knowledge of Cabal internals and would probably be sufficient for most uses.)

[ezyang]

@BardurArantsson I'm attempting to write a test with your new format, and I don't understand, how are you supposed to actually get the intended output? Is there an make accept option like in GHC? I don't see any code which would support that.