Plan for better supporting multi-component repls

[mpickering]

Right now there is a limitation to cabal which means that you have to build all a packages dependencies before configuring a package. When combined with multiple home units this is a serious issue because you have to build all the packages first before you start your multi-component session which then builds them again. This limitation makes multi-component unsuitable as a means of doing builds for example (as much work is duplicated).

In addition, for clients such as HLS, which has support for multi-component sessions, having to build everything before discovering the options is sub-optimal (for the same reason, the build will be completed again by HLS).

So it would be good to find a way to configure all the packages we're going to build before we actually build any of them. There are two problems to overcome:

Problem 1: Setup.hs scripts might depends on the assumption that all dependencies are built (for example might consult the package database to look at the abi of a dependency).
Problem 2: The Setup.hs configure logic requires that a package has been built (and in the package database) for the configure step to work.

The proposed solution to these issues is:

1. Add a new custom Setup.hs type which distinguishes whether the Setup.hs requires the dependencies to be already built or not before configuring. Almost all custom Setup.hs do not require dependencies to have been built.
2. Modify the configure logic so that it's possible to distinguish between packages which should be in the package database and packages which are going to be built together. Therefore it's possible to configure against a mix of installed and "virtual" packages which are not yet built (but will be built together in a multi-component session).

[Mikolaj]

What is a multi-component session? Could you link to some docs? What else uses it, except HLS? Are there any simplified examples of its use around?

We are very slowly (as in, in future decades) moving towards deprecating custom setups, so would it be possible to place the NO_DEPS_REQUIRED flags somewhere else?

From a cabal dev's perspective, it's a worthy goal to understand and document why deps need to be built before setup-configure and generalize the cabal machinery by relaxing that requirement. This may be a good place to start and we'd welcome docs and refactorings it could bring.

Independently, let's discuss the requirements for "multi-component repls" and how to best satisfy them.

[lf-]

Hi! I'm currently writing a blog post on a workaround for the absence of this feature so I have lots of context on this.

The core cabal-relevant issue here is that cabal only deals with one target at a time, the so-called "home unit", following the former limitations of GHC. For instance, cabal repl cannot load multiple targets, which is frustrating while working on the executable and the library at the same time, or the test suite and the library at the same time since :reload won't work properly.

You cannot cabal repl test:sample lib:sample, for instance. Stack does support stack ghci exe:sample lib:sample, but it's achieved through constructing a fake target that merges the GHC options for both targets, but this is a hack and has several limitations. The post I'm writing is on doing basically this but in a cabal file instead.

GHC now supports the real fix for the multiple home units issue upstream, for which Well-Typed has written a nice blog post. There is also the original GSoC retrospective post from Fendor which provides more context.

The current situation, as I understand it, is that GHC supports multiple home units, but cabal does not do the invocations or data model necessary to support it. In the GSoC retrospective linked above, the author mentions having seriously hacked up cabal, breaking the boundary between cabal-install and Cabal, in order to test this, due to some incompatibilities between the data model of cabal and having multiple home units. That patch is obviously not mergeable, and the cabal model would have to be adjusted more carefully/correctly.

Thus my understanding of the scope of the work on this problem is to improve the data model for cabal to understand multiple home units and invoke GHC accordingly.

Another piece of outside context I can offer is that people have been working on making Bazel work for Haskell, to some extent since the combination of Cabal/GHC don't yet know how to do cross-package build parallelism yet (for instance, say that package A depends on package B, but not all of A depends on B, so the modules of A that don't need B build in parallel): https://www.tweag.io/blog/2022-06-23-haskell-module/

This issue is relevant to this (possibly a duplicate even!): #8238

I understand that @wz1000 is working on parts of this currently.

[mpickering]

@dcoutts pointed out that if a Setup.hs depends on a certain package being built already then this can be expressed by adding that package to the build-depends of the Setup.hs.. of course this is a bit janky because you do not require the package for the build phase but rather the run phase but I think it's a more pragmatic solution than a new custom Setup.hs type.

@lf- summarises the situation quite well :) Thanks for writing that up. If you are interested I also implemented the cross-package parallelism idea in hadrian.

[parsonsmatt]

I would absolutely love to have support for cross-package parallelism. That would make it much easier to break things into separate packages in our Large Work Repo without negatively harming compilation time.

[Mikolaj]

Anything is better than a new custom Setup.hs type. :) Please feel free to mock up a PR.