RFC: design for better GHC plugin support in cabal

[michaelpj]

This came out of some discussion on IRC. I started writing an issue, but on reflection I think there's a relatively plausible design sitting there, so I wrote it down.

Motivation

GHC compiler plugins are currently not terribly easy to use with cabal, and require some amount of manual fiddling. In addition, they are not specified as declaratively as they could be (i.e. cabal has no concept of a "plugin"), which prevents cabal from doing some things as well as it could do.

I think there are two broad kinds of plugin that we should be thinking about:

1. Package plugins

Package plugins are those which are required by the package itself. They usually have a load-bearing role, in that they are actually necessary to compile the package. An example is the record-dot-preprocessor plugin.

Today, these can be used by:

• Adding the plugin package to the component's build-depends stanza
• Passing the appropriate -fplugin GHC option

Package plugins as implemented today have a few weaknesses:

• They don't use -fplugin-package, since they are just specified as normal build-depends. This means they get unnecessarily linked into the final build product (Add support for -fplugin-package #6169).
• They are solved for as part of the main build plan, even though (see above) they don't need to be linked into the final build product.
• If/when the upstream GHC support for a native package db for plugins is completed, cabal will not be able to use it (because it doesn't know they're plugins).
• Users have to specify the -fplugin options themselves, as well as any -fplugin-opt flags.

2. Global plugins

Global plugins are ones which users want to run on all packages that get built, regardless of whether the package itself says it requires any plugins. An example is the ghc-tags-plugin plugin. Often these are used to e.g. get additional compile-time information out of dependencies.

Today these are very difficult to use (#7685, #6307, probably more). Partly the issue is that doing this nicely requires both setting a GHC option for upstream packages (doable today in cabal.project), but also building the plugin package and ensuring it is present in the appropriate package db when building the upstream package (hard to do without hacks like specifying your own manually created plugin db).

Since these sorts of plugins are also frequently used just in development, it would be particularly nice if they could easily be specified via cabal.project.local (#6169 (comment)).

Proposal

1. plugin-depends field for .cabal files

(Prior art: #2965, but I'm putting it a little differently, so I'll repeat.)

The goal here is to make the situation nicer for package plugins. To that end, we teach cabal a plugin-depends field for component stanzas for .cabal files. plugin-depends contains a list of package/module pairs.

Initially, it could start out as a simple wrapper for build-depends, and then we could progressively augment it to have the following features:

• Also imply the appropriate -fplugin flags. (This is why we need the module component! -fplugin needs the actual plugin module.)
• Be passed via -fplugin-package (in particular, I don't think this needs to be a prerequisite, contra RFE: local plugins and build-tool-depends #6307 (comment))
• Be included in an appropriate native package db when/if that exists
• Be solved for as part of an independent build plan for native dependencies (relies on the previous item)

This deliberately doesn't cover -fplugin-opt flags. It seems reasonable for users to add those themselves.

(I'm also disagreeing with my past self here, who thought that we should just have native-build-depends and have that include plugins. However, the idea of including the extra information about modules that's needed to automatically pass -fplugin flags seems appealing to me, and requires a plugin-specific field. You could do this with a combination of native-build-depends and active-plugins (the latter taking a list of modules); maybe that would be nicer.)

2. plugin-depends package field for cabal.project files

The goal here is to make the situation nicer for global plugins. To that end, we teach cabal a plugin-depends field for package stanzas in cabal.project files.

This field simply augments the plugin-depends field for the corresponding package. Plugins that truly want to be globally applied can then use package * as usual.

[phadej]

plugin-depends having package name and module names will be messy. What if package provides multiple plugins etc. Just keep them separate.

Sometimes plugins need to be applied only in a handful of modules.
That's another reason why plugin-and-splices-depends and ghc-plugins should be separate.

plugin-depends is/could be generalized to ghc-proposals/ghc-proposals#243 and/or ghc-proposals/ghc-proposals#412, i.e. TH dependencies.

Dependencies needed for TH would come from the same "host" dependencies.

-- I hope this name is awful enough it won't stick
plugin-and-splices-depends: overloaded, record-dot-preprocessor, my-awesome-type-class-th

-- Naturals is the option passed to overloaded.  Token syntax can be used here.
ghc-plugins: Overloaded Naturals, RecordDotPreprocessor

-- overloaded needs to be dependent upon, because it provides both the plugin and the library stuff.
-- there is no good reason to separate these today.
build-depends: overloaded

Existence of ghc-plugins could imply that compiler is ghc (and/or ghcjs?) for solver.
Maybe even plugin-and-splices-depends too, as there isn't other compilers with TH.

[Mikolaj]

If this could solve https://gitlab.haskell.org/ghc/ghc/-/merge_requests/5965, that would be indeed high priority to implement.

@bgamari

[michaelpj]

If this could solve https://gitlab.haskell.org/ghc/ghc/-/merge_requests/5965, that would be indeed high priority to implement.

I don't think it would do that?

[gbaz]

@michaelpj do you still support this? I saw this linked from a newer issue, and I think this is a cleaner approach we should just do. I'm not sure I understand the concerns oleg raised, and I'm not sure they're significant enough to warrant against this simpler approach.

If you think this simpler approach is still valid, and still worthwhile, I think we would accept a patch for it.

[gbaz]

Hrm -- actually, we would want plugin-depends to alias build-tool-depends or setup-depends maybe, not build-depends, right? Then we get "independent solving" for free.

Or er... not sure actually. build-depends is certainly the easiest first step.

[michaelpj]

I do think this is nice. I continue to think we probably want a specific plugin-depends field given the existence of -fplugin-package and so on. I'm fairly convinced by Oleg that we should just leave it as the package dependencies and handle the specification of "active plugins" separately or not at all (let the user set -fplugin themselves). So plugin-depends would just be the package dependency.

It would be nice to have some opinions from other knowledgeable people, perhaps @angerman ?

If you think this simpler approach is still valid, and still worthwhile, I think we would accept a patch for it.

This isn't going to get to the top of my list soon, I'm afraid...

Hrm -- actually, we would want plugin-depends to alias build-tool-depends or setup-depends maybe, not build-depends, right? Then we get "independent solving" for free.

I don't really know enough about the innards of cabal to say what would be the easiest implementation strategy 🤔

[angerman]

The core of this is in my opinion that we end up with two separate package databases; that have distinctly different purposes. plugin-depends is decidedly not build-depends. It's a runtime dependency for the build process. These should go into a separate package-db. I am against having auto stuff for -fplugin, if someone want's to use a plugin, use ghc-options for this coupled with plugin-depends. If you depend on the plugin, and the plugin also depends on a build-time library, you'll have to add it to both build-depends and plugin-depends. This is IMO the right approach.
plugin-depends is much closer in spirit to setup-depends.

In a hypothetical future where cabal properly understands cross compilers, this will become necessary anyway, as the setup-depends/plugin-depends will have host native code, and the build-depends target native code.

The alternative would be to eventually have ghc package databases be multi-target aware, and then lump it all together into one package-db, but that seems a lot more messy than just cleanly separating package-dbs for their respective purpose.