Interface for system package manager to execute build plan

[Ericson2314]

Original Post

I wrote a fairly long email to the nix-dev list (http://lists.science.uu.nl/pipermail/nix-dev/2016-September/021765.html) proposing that cabal2nix work from cabal-install's build plans instead of the package cabal files themsevles. This would obviated needing to teach it to resolve configuring options etc, basically replicating Cabal. [The idea is of cabal-install resolving the plan and the system package manager executing the plan generalizes for any system package manager, not just Nix, hence not singling Nix out in the issue title.]

Reading the cabal-install section of the componentized Cabal proposal leads me to further propose that plan.json (or whatever should be the interface between the system package manager and cabal-install) contain exactly the component dependency graph proposed in the RFC. This granularity would be nice for Nix for the same reason it is nice for cabal and stack---I personally will contest that playing around with some packages and then having to rebuild them to enable tests is quite annoying.

A final extension to that would be to extend Cabal itself to also work from these a single-component manifests. The default Setup binary would gain the ability to work from either a manifest--corresponding to a single component, or a bunch of manifests--corresponding to that component and its dependency closure. [I say either because I am not sure whether the new-* work affect Cabal or just cabal-install. If Cabal too, I assume the closure is needed, if just Cabal then the single component's manifest is probably fine.]

There are two advantages of this final bit. The first is that we currently only use Cabal, not cabal-install, in Nix Haskell packages, and this would allow us to continue doing that. That said, I'm not sure what the advantages of this current approach so this may not be good motivation. The second is were we also wanted to use stack to generate build plans, the component dependency dag is probably a good format for both to support (I speculate given that the componentization proposal was originally @snoyberg's idea). stack --nix already exists but for reasons given in my mailing list post, I think the mechanism I'm describing would work strictly better.

Anyways, after writing the proposal for the Nix list, I thought it would be good to get the attention and opinion of upstream.

Current Plan for Nix

For non-local dev (packaging executables for nixpkgs), here is a sketch of the entire process:

1. Make the cabal.project
   • Use extra-packages: for the executables, since they come from hackage
   • Use compiler: to specify the compiler so we don't it built at config time.
   • Ignore native deps
2. cabal new-configure without any network access and limited FS access only to cabal-install's runtime deps + downloaded index
   • Neither the compiler, executable deps, or native library deps need to be built/exposed at this time.
   • TODO some way to give cabal-install a nix-downloaded index, ideally without weakening with the TUF stuff. Maybe .cabal/config already handles this? I assume we can put the config elsewhere with a CLI flag?
3. Resulting plan.json contains "install items" each with:
   • Basic metadata like package name, component name, and version
   • Relevant calls to ./Setup with all flags
   • TODO: pkg-config deps relevant to just this component. (./Setup configure is still responsible for checking whether those libraries are present, but cabal configure can anticipate when they will be looked-for.)
   • TODO: Deps on other "install items"
   • TODO: hackage urls + hash sufficient for Nix to fetch
4. Turn plan.json into a package set where each "install item" becomes a derivation. Either this comparatively simple task is the new job of cabal2nix, or this is so simple we can just use builtins.fromJSON and write in Nix.
   • Use Nix-style uniq ID's to replicate dependency graph structure.
   • Fetch and unpack Hackage tarballs.
   • Put the./Setup commands specified in the proper phases. Hopefully we pass cabal new-configure enough so that the ./Setup flags do not need to be modified.

Work to be done

• build Setup.hs with local compiler when cross-compiling #1493: host vs target tools

[23Skidoo]

/cc @ttuegel

[23Skidoo]

Related: #3651.

[Ericson2314]

So it's my opinion that (to borrow the git terminology) stack --nix implements porcelain without the plumbing. I.e, it looks nice but in my experience doesn't really work properly outside of more trivial scenarios, so this isn't a purely theoretic design disagreement.
What I'm proposing here is strictly not Nix-specific, but IMO gives Nix or any other auxiliary package manager the interface needed to do things right. After this is done, I think the time would be ripe to do a more slick CLI like stack --nix. With this plan, the implementation of #3651 would be quite different, but if we go this route we can redesign that when we get there.

[ezyang]

I wrote a fairly long email to the nix-dev list (http://lists.science.uu.nl/pipermail/nix-dev/2016-September/021765.html) proposing that cabal2nix work from cabal-install's build plans instead of the package cabal files themsevles.

That sounds fine, but it does mean we have to commit to a more stable and documented plan.json format. (We still don't know if we've put enough stuff in yet, if it actually is usable by people ;) It also means we have to commit to a specific interface for configuring and building packages. That also should be fine; the Cabal command line API moves very slowly.

A final extension to that would be to extend Cabal itself to also work from these a single-component manifests.

Are you sure you want to add a new single component manifest format, as opposed to plan.json recording a pile of configure/build flags/arguments that you need to use? That would be more backwards compatible and you could start using it today. And as I said above, the configure/build API is pretty stable.

Next we instruct cabal-install to decompose the plan into "sub plans" for each dependency/would-be system package---this is the biggest change needed for cabal-install, but one that any system package manager could leverage, not just us.

I don't know what you mean by this. Could you maybe write some concrete JSON pseudocode describing what a sub plan looks like?

Finally, each of those derivations can use cabal new-install to install their root package

Did you mean cabal install, or did you mean calling the Setup script above (which is what you mentioned earlier?)

since the Nix derivations will also depend on each other, cabal new-build's own caching should kick in so each Nix derivation does in fact correspond to each Haskell package being built.

new-build caching works in a very specific way: for any specific unit identifier we depend on, we look in .cabal/store to see if the directory for that library already exists (and it is in the package database). If so we improve into the installed version. Is that interface sufficient for Nix or do you need something else.

I didn't read the section on local dev with Nix, I know very little about how it works.

[ezyang]

@Ericson2314 are you interested in contributing and maintaining the necessary patches to cabal-install to make this work? I think it would be much appreciated by the Nix community.

[Ericson2314]

[On subway so limited Internet.] I'd hope since this isn't Nix-specific I wouldn't be solely on the hook for maintenance :), but I'm certainly down to implement.

[ttuegel]

Despite appearances, #3651 is actually orthogonal to this issue. What is proposed here is really an alternative to cabal2nix, which itself produces a poor approximation of a build plan. Even with #3651, Nix users would benefit from this proposal. And, even with this proposal Nix users would need #3651 to manage non-Haskell dependencies, which is actually my sole motivation for implementing it.

I don't think this proposal offers any features beyond what new-build does, just a different mechanism.

[ttuegel]

I want to single-out one statement from the mail to nix-dev:

During this stage it in principle only needs access to .cabal files / the Hackage index and a project config---certainly not any already built binaries like today.

The build-type: Custom interface makes this untrue, unfortunately.

[Ericson2314]

but it does mean we have to commit to a more stable and documented plan.json format. (We still don't know if we've put enough stuff in yet, if it actually is usable by people ;)

Well, if there is a helper executable to call GHC etc on behalf of the system package (e.g. stock ./Setup as I tentatively proposed only in this thread), then plan.json can still be somewhat of a black box.

It also means we have to commit to a specific interface for configuring and building packages. That also should be fine; the Cabal command line API moves very slowly.

Because all the configuration should be done beforehand by cabal-install, the CLI for this should be very simple and need not look nice so as to not sqat on valuable CLI real-estate should it need to be deprecated.

Did you mean cabal install, or did you mean calling the Setup script above (which is what you mentioned earlier?)

In the mailing list, I erred on the side referring to cabal-install, as I am not sure quite what the division of labor is for new-build. Here, I went on a limb and proposed using ./Setup, hoping Cabal is enough. Sorry for the inconsistency + confusion.

I didn't read the section on local dev with Nix, I know very little about how it works.

That is icing on the cake / more #3651's territory, so feel free to ignore it for now :).

I don't know what you mean by this. Could you maybe write some concrete JSON pseudocode describing what a sub plan looks like?

new-build caching works in a very specific way: for any specific unit identifier we depend on, we look in .cabal/store to see if the directory for that library already exists (and it is in the package database). If so we improve into the installed version. Is that interface sufficient for Nix or do you need something else.

Well, to get the easy concerns out of the way, we would have one new-style .cabal/store per system package, analogous to what we do with GHC_PACKAGE_PATH and legacy DBs today.

Now the complex bit. Well, what, if anything in .dist-new is used? Ignoring local builds for now, that directory will not be accessible for the easiest method on Nix's end, or we can rebuild part of it. The key issue is that Nix will be more conservative about hashing, and so we don't want irrelevant parts of .dist-new affecting the current build.

Extrapolating from Nix, I assume building needs more than the .cabal/store and the current *.cabal because then we'd be back to square one configuring plans biased by what's been built---and doing so per package/component not per cabal build which is even worse. So parts of the plan are needed. Now packages that aren't a (transitive) dependency can definitely be pruned from the plan so as not to trip of Nix's conservative hashing. The question remains do we need just the plan for the current package/component, or also the plan for each of it's transitive dependencies? Ideally, since the current plan includes the unique identifier (with hash), enough metadata can be put in Cabal(-install)'s store so only the JSON object (/binary equivalent) for the current package/component is needed.

[One final hiccup is I'd like not to have all non-haskell deps build and exposed at cabal new-configure-time, but if deferring that part of configuring until the packages/components that need them are being built is hard, let's forget about that for the first version of this.]

[Ericson2314]

@ttuegel Hmm? it was my understanding that we pray assume that custom setups don't affect the build plan, so cabal2nix certainly ignores them. Does cabal new-configure not?

[Ericson2314]

[edit pipeline sketch is now at the top of this thread]

[Ericson2314]

CC @peti as this is more active than the nix-dev thread.