Support non-Hackage dependencies

[23Skidoo]

Summary by @ezyang:

• Syntactic support for these URLs exists: in the packages field you can specify remote tarballs or local tarballs, and using the source-repository syntax (from Cabal file) you can point to a VCS URL (this parser should also have been reused for cabal.project)
• Support for downloading from various places exists. There's support in the old-build path for local/remote tarballs. cabal get -s supports getting things from source repositories.
• However, these are not used for anything: the internals need to be plumbed up to use this. Look at old stuff and determine how to port it over.

This should be doable in a day.

---

This came up on Twitter:

https://twitter.com/darinmorrison/status/527716059889954816

Basically, people want to publish packages that depend on stuff that is not on Hackage. One solution is to upload a forked package (example), but this has its own downsides (Hackage namespace becomes polluted). Implementing something like build-depends: https://repo.git@revision would be a nice alternative. Most other package managers (e.g. sbt) support this.

It should also be possible to prohibit downloading packages not on Hackage via a config file option.

See also #1534.

[mietek]

Halcyon supports declaring sandbox sources with the HALCYON_SANDBOX_SOURCES option.

See Try Haskell for an example of declaring the git HEAD version of mueval as a sandbox source.

[winterland1989]

I'd like to bump this issue's priority and start to discuss implementation, currently it's stopping us using cabal in company environment because we have multiple packages can't be upload to hackage, and a hackage mirror is an overkill. now we use stack to solve this headache, but it's really not necessary to rely both on hackage and stackage infrastructure.

[ezyang]

Hello @winterland1989. Have you tried the new new-build functionality (http://blog.ezyang.com/2016/05/announcing-cabal-new-build-nix-style-local-builds/). If you have all of the non-Hackagable packages on disk in a build tree, you can write a cabal.project to pick them up, and then cabal new-build will automatically build them together. Does that solve your proximal problem?

[winterland1989]

Thanks for your reply @ezyang ! I'm aware that there're multiple ways solving local package discovery: cabal sandbox add-source or a cabal.project file both solve this, and i'm aware that cabal.project file support parsing URI as package now. But we need an automatic way to pull/update the remote file into disk without hassle, currently available options are git submodule, hand-role shell and stack.(you really want to automate this process when you got more than 5 local depends).

If cabal.project file is the final solution, i'll dig into it to see what i can do.

[23Skidoo]

@winterland1989
There is already some code for declaring VCS dependencies in cabal.project files: https://github.com/haskell/cabal/blob/master/cabal-install/Distribution/Client/ProjectConfig.hs#L523

If you want to contribute code to Cabal, I suggest improving the nix-local-build code path, since the plan is to make it the default in the future.

[alanz]

I would like to see something like http://docs.haskellstack.org/en/stable/yaml_configuration/#complex-package-locations-location

[phadej]

Looks like we only need to teach cabal-install how to fetch stuff from the internet:

cabal/cabal-install/Distribution/Client/ProjectConfig.hs

Line 745 in 5423d77

fail $ "TODO: add support for fetching and reading local tarballs, remote "

tar support shouldn't be difficult, as cabal handles it already. git is trickier, as we need to teach cabal-install to call git clone (and darcs).

Question: were to put downloaded stuff, dist-newstyle/downloaded/hash-of-the-source ?

[phadej]

(For now one can use git submodule, effectively the result should be the same)