source-repository-package not optional

[nomeata]

I was hoping that I can use source-repository-package to point to “optional” dependencies, i.e. dependencies that are used when needed, but not built when not. I want to use this in my cabal.project:

source-repository-package
      type:     git
      location: https://github.com/ghcjs/ghcjs-base
      tag: 01014ade3f8f5ae677df192d7c2a208bd795b96c

because ghcjs-base is not on Hackage yet. The problem is that this package is only a dependency of jsaddle when using ghcjs, but not when using ghc. Furthemore, ghcjs-base has dependencies that cannot be fulfilled with ghc.

So with this in cabal.project, cabal thinks it has to build ghcjs-base, even with using ghc, and fails.

[typedrat]

So basically, they shouldn't be added like they're another package in the same project, but they should just expand the universe of things that the solver knows about to depend on?

[nomeata]

At least that was my (possibly naive) expectation. The main reason I want to use this feature is to be able to build packages that are not being uploaded to Hackage, like reflex (reflex-frp/reflex-dom#140, reflex-frp/reflex#177)

[typedrat]

That feels like how it should work to me, too.

[hvr]

This is a general problem; we only have packages: and optional-packages: available; which force packages to become local ones which are solved for (even if they're not necessarily built). So source-repository-packages are basically like having Git submodules + packages:. From that POV, that's consistent.

Otoh, we can add additional package repo indices via repository-stanzas; these would just "expand the universe" w/o them being included always in the build-plan.

What would be useful to have is a way to merely add single packages to the "universe" (similiar to repository-stanzas; but much simpler), and then have a property-field for source-repository-packages in order to declare whether this package ought to be local (like it is now), non-local (i.e. cached in the nix-style store), or index (i.e. merely "added to the universe"; as if it was included in a package repo index).

PS: @nomeata for now you can workaround this by having separate cabal.project files for ghc and ghcjs; you can then use --project-file= to select the "personality" you need.

[gbaz]

That idea (adding packages to the universe) was basically the cabal-side design of the provenance-qualified package imports proposal: ghc-proposals/ghc-proposals#115

Cabal could adopt that or something similar even without GHC actually implementing the ability to track provenance in the packagedb proper.

[nomeata]

The documented use case for `optional-packages`` is vendored dependencies. I guess usually you use your dependencies, so usually people don’t notice the difference, but I would argue that the current behaviour is not following the principle of least surprise.

PS: @nomeata for now you can workaround this by having separate cabal.project files for ghc and ghcjs; you can then use --project-file= to select the "personality" you need.

Doing that already :-)

[hvr]

@nomeata well, the main use-case for source-repository-package as I see it is to cause a specific snapshot of a package to be used in your install-plan; so if you point your cabal.project file to some git repo, I'd argue the principle of least surprise would be in fact require that the Git version is taken, and not some other version that might have been on Hackage.

If we now wanted the variant that such (possibly temporarily) vendored dependencies are declared "exclusive" (i.e. they hide all other versions for the same package-name), but at the same time act like index-packages (i.e. they're only included in the install-plan if anything force them in by actually depending on them), we need yet another mode (strawman: index-exclusive) than the one I suggested in #5444 (comment)

(I'm not sure how provenance-qualification would benefit here; once remote git repo deps are involved, we're not in a self-describing package context anymore, but instead in a cabal.project-specific context where the cabal.project file sets up the "routing" of where packages are procured from -- i.e. that's where the "local" of "nix-style local builds" becomes apparent: we're dealing with local versions of packages which are supposed to supplant/shadow any pre-existing Hackage packages of the same name; I don't think we intend to introduce new names or labelled namespaces when vendor a package in this context -- for example, if I locally vendor a Git snapshot of text, I want that version to be used everywhere across my install-plan, and w/o having to modify any build-depends; nor do I think I care about the provenance information if the pkg is registered in a nix-style pkg store/cache or even merely the local dist-newstyle pkg-db as I don't see what this information would be useful for)

[nomeata]

I'd argue the principle of least surprise would be in fact require that the Git version is taken, and not some other version that might have been on Hackage.

Good point…

index-exlusive sounds right. Not sure if we need all the others :-)

[hvr]

well, you need the local one at least to mark your "primary" local packages; otherwise cabal new-build all would not build anything

PS: we also have extra-packages: which I didn't bring up yet...

and here's a link to the docs for convenience: http://cabal.readthedocs.io/en/latest/nix-local-build.html#specifying-the-local-packages

[nomeata]

The docs don't help me to understand why I'd want extra-packages.

[gbaz]

@hvr, you write

I'm not sure how provenance-qualification would benefit here; once remote git repo deps are involved, we're not in a self-describing package context anymore, but instead in a cabal.project-specific context

But the provenance quantification proposal described how to specify git repo deps as repositories, which is exactly what you're describing. And it described how to do so within cabal files rather than just project files.

Cabal files are extended in two ways.

source-origin-alises: is a top-level property consisting of a list of aliases in the usual indented assignment style. These aliases will be passed to all invocations of ghc in the course of a cabal build. Example syntax:

source-origin-aliases:
hackage: https://…
foobar: http://…

build-depends: fields now can take package names in the form of [REPO]+somepackage rather than just somepackage.

All that is required is to allow source-origin-aliases to take aliases to git repos of single packages as well as standard repositories, which can be derived directly from the syntax we already have for source-repository-package.

[hvr]

(I briefly talked with @gbaz about the provenance qualification; there's overlap in what you can do via cabal.project already now and what prov-qualification would allow, but the intentions of the two mechanisms diverge)

@nomeata that's true; the documentation of extra-packages is very lacking; you can read however Duncan's rationale for it at e.g. #3737 (comment) and a recent enhancement to its syntax (which also hasn't found its way into the docs) was merged via #4889

[nomeata]

This discussion prompted me to not use source-repository-package for my goal, but rather set up my own hackage overlay repository with these packages:
http://hackage-ghcjs-overlay.nomeata.de/
https://github.com/nomeata/hackage-ghcjs-overlay
A bit manual, but makes the travis scripts that uses them easier.

[hasufell]

What needs to be done to move this ticket/the discussion further? This is still a problem today.

[Fuuzetsu]

Another problem that this causes: all source-repository-packages are local and are therefore registered as -inplace rather than have some UnitID. This means tools like cabal-cache aren't able to cache these which is very sad: we have maybe some 30+ packages listed like this.

The problem is greatly amplified in that it's enough for a single package deep down in the transitive chain to completely mess up caching. For example, we're using a custom commit of the socks library. That now gets registered as local which means everything that depends on that (like http-client-tls and then amazonka and everything upstream from there) is now -inplace and can't be cached. That's quite said. I believe in stack this is exactly what extra-deps field achieves FWIW.

EDIT:

As follow-up for anyone looking in future: it appears that you can leave cabal-cache to downloading anything from the cabal store and everything left over in dist-newstyle should just be in-place dependencies. You can have your CI cache this bit in naive way.

[Fuuzetsu]

I believe in stack this is exactly what extra-deps field achieves FWIW.

I think I was mistaken actually: https://stackoverflow.com/questions/54351314/how-to-cache-a-haskell-dependency-to-speed-up-compilation-time

[phadej]

@Fuuzetsu

Another problem that this causes: all source-repository-packages are local

Have you tried cabal-install-3.4 release candidates?

E.g. in https://github.com/phadej/trustee which has two source-repository-package definitions I see

UnitId "peura-0.20201018-d5fb116a42702e0c31cde71ced2a98aed4f889bb86bd9ab549d93547250bf689"
UnitId "urakka-0.1-73d82c94439d9c8b493f147de03fa28ce2e4ef7a15389feb17d9595fe0dd7378"

in cabal-plan plan.json dump.

[hasufell]

@phadej 3.4 is indeed an improvement, but afais the solver still considers all source-repository-packages, even if it isn't actually part of the depgraph (e.g. because it's behind a disabled flag or because it isn't an actual dependency of anything).

As in: in some projects I have to remove a source-repository-package from cabal.project to make the project build (examples are cardano-wallet/cardano-node with systemd disabled).

[Fuuzetsu]

Have you tried cabal-install-3.4 release candidates?

I tried it and it seems all the inplace packages and their dependencies now have proper unit IDs and can be picked up via cabal-cache, that's great. That solves half of my problem. The rest of the issue is just what this ticket is about.

[gbaz]

I think allowing conditionals in project files (that can case on compilers, etc just like in packages) would resolve this as per #7783 ?

[hasufell]

I think allowing conditionals in project files (that can case on compilers, etc just like in packages) would resolve this as per #7783 ?

I think that's just a workaround. Local packages that are not needed for the build and are not requested as targets by the user shouldn't be part of the resolution.

[phadej]

Local packages that are not needed for the build and are not requested as targets by the user shouldn't be part of the resolution.

That would ruin install plan caching e.g. in project where there are different packages:

cabal build pkg1
cabal build pkg2

would need to run solver in between, which will make cabal feel non-interactive. If both packages are built, those commands should be no-op and fast (<10ms).

[hasufell]

That would ruin install plan caching e.g. in project where there are different packages:

I'm not sure I follow. I'd expect Cabal to cache plans for all the targets separately then, unless there's a build plan that works for all of them.

I also think your described behavior is too implicit and not a good default.

I'd actually expect two resolutions, unless I did run something like cabal configure all beforehand.

[phadej]

I'd expect Cabal to cache plans for all the targets separately then,

That would be another caching strategy, cabal-install doesn't do that now, but rather follows single install-plan strategy. Implicit enabling of some targets in cabal test and cabal benchmark is weird though. But as nothing related is documented nor specified, I guess it all can be changed (if someone specifies the precise semantics so we don't need to argue what is a feature and what is a bug, until then I'd consider current behavior to be indented feature).

[phadej]

I'd welcome a feature where tar.gz local and remote packages, and source-repository-packages were treated like override package-repositories (i.e. like a single version "hackage"-like repositories) rather than a must have packages.
i.e. if package in source-repository-package is needed, than it's build, but cabal build all won't match on it.

However, it would be weird to say

packages: http://.../foo.tar.gz

and cabal build all won't build foo, because foo is in packages:. In fact, stack had a way to specify git packages in their packages:, but later moved them to extra-deps (which are more like package-repositories specification, though not exactly). So it would be better to have different field for these to avoid possible confusion.

EDIT: currently e.g. --enable-tests doesn't enable tests in source-repository-package as they are not "local", but they are still project packages. So they are something in between, and that something in between is also confusing.

That is a somewhat radical change (i.e. only truly local unpacked packages were "project" / "local" packages), which needs some dedicated person to push through. I won't be one.

[ParetoOptimalDev]

source-repository-packages were treated like override package-repositories

Can anyone point me to the documentation (if it exists, I couldn't find it) or implementation for "override package-repositories"?

[Mikolaj]

I don't know, but somewhat related is #7783.