Dependency on installed public sublibrary without --dependency flag

[fgaz]

follow-up from #5660 (comment)

Depending on a globally installed public sublibrary is problematic unless --dependency=pkgname:sublibname=installed-id is specified.

The bug (?) does not get triggered when that flag is used, for example:

• when manually running Setup configure --dependency
• when running cabal-install on source packages

Here are two occasions in which it can manifest (b depends on a:foo):

In nix, which I think does not use --dependency and instead relies on Cabal's basic solver, I get this error:

build log

building '/nix/store/djsi6bxhgwgqlvymi3rf5078567jry3w-b-1.drv'...
setupCompilerEnvironmentPhase
Build with /nix/store/2lkbmrf9iw3v4dnf0yixhldvff5397xl-ghc-8.10.3.
unpacking sources
unpacking source archive /nix/store/0h1n08ri9fph049mx0rxxx8qnl4pz408-b
source root is b
patching sources
compileBuildDriverPhase
setupCompileFlags: -package-db=/build/setup-package.conf.d -j4 +RTS -A64M -RTS -threaded -rtsopts
[1 of 1] Compiling Main             ( /nix/store/4mdp8nhyfddh7bllbi7xszz7k9955n79-Setup.hs, /build/Main.o )
Linking Setup ...
configuring
configureFlags: --verbose --prefix=/nix/store/pl31hr7cnig3nfqak49nkgiivnphddm9-b-1 --libdir=$prefix/lib/$compiler --libsubdir=$abi/$libname --with-gcc=gcc --package-db=/build/package.conf.d --ghc-options=-j4 +RTS -A64M -RTS --disable-split-objs --enable-library-profiling --profiling-detail=exported-functions --disable-profiling --enable-shared --disable-coverage --enable-static --disable-executable-dynamic --enable-tests --disable-benchmarks --enable-library-vanilla --disable-library-for-ghci --ghc-option=-split-sections --extra-lib-dirs=/nix/store/8v0snmd618lmk6w3xzapf4ygac1n8adx-ncurses-6.2/lib --extra-lib-dirs=/nix/store/8b4r4lmhybjxhxx33z0si8gifhi8hfmn-libffi-3.3/lib --extra-lib-dirs=/nix/store/smhc5diparm6im6dp3iqbz460hwrlv8s-gmp-6.2.1/lib
Using Parsec parser
Configuring b-1...
Warning: Packages using 'cabal-version: >= 1.10' must specify the
'default-language' field for each component (e.g. Haskell98 or Haskell2010).
If a component uses different languages in different modules then list the
other ones in the 'other-languages' field.
Dependency a:{foo} -any: using a-1
Dependency base -any: using base-4.14.1.0
Source component graph: component lib:bar
CallStack (from HasCallStack):
  dieNoWrap, called at libraries/Cabal/Cabal/Distribution/Utils/LogProgress.hs:61:9 in Cabal-3.2.1.0:Distribution.Utils.LogProgress
Error: Dependency on unbuildable library 'foo' from a

builder for '/nix/store/djsi6bxhgwgqlvymi3rf5078567jry3w-b-1.drv' failed with exit code 1
error: build of '/nix/store/djsi6bxhgwgqlvymi3rf5078567jry3w-b-1.drv' failed

in cabal-install: when depending on a globally installed package (for which the .cabal is not available)

build log

> cabal configure -v3
File monitor 'config' changed: first run
Project settings changed, reconfiguring...
creating /home/build/p/b/dist-newstyle
creating /home/build/p/b/dist-newstyle/cache
this build was affected by the following (project) config files:
File monitor 'improved-plan' changed: first run
File monitor 'elaborated-plan' changed: first run
File monitor 'compiler' changed: first run
Compiler settings changed, reconfiguring...
Searching for ghc in path.
Found ghc at /opt/ghc/bin/ghc
/opt/ghc/bin/ghc --numeric-version
/opt/ghc/bin/ghc is version 8.10.1
looking for tool ghc-pkg near compiler in /opt/ghc/bin
candidate locations:
["/opt/ghc/bin/ghc-pkg","/opt/ghc/8.10.1/bin/ghc-pkg-ghc-8.10.1","/opt/ghc/8.10.1/bin/ghc-pkg-8.10.1","/opt/ghc/8.10.1/bin/ghc-pkg"]
found ghc-pkg in /opt/ghc/bin/ghc-pkg
/opt/ghc/bin/ghc-pkg --version
/opt/ghc/bin/ghc-pkg is version 8.10.1
/opt/ghc/bin/ghc --supported-languages
/opt/ghc/bin/ghc --info
File monitor 'solver-plan' changed: first run
/opt/ghc/bin/ghc --print-global-package-db
Reading installed packages...
/opt/ghc/bin/ghc-pkg dump --global -v0
/opt/ghc/bin/ghc --print-libdir
Reading available packages of hackage.haskell.org...
Using most recent state (could not read timestamp file)
Warning: The package list for 'hackage.haskell.org' does not exist. Run 'cabal
update' to download it.RemoteRepo {remoteRepoName = "hackage.haskell.org",
remoteRepoURI = http://hackage.haskell.org/, remoteRepoSecure = Just True,
remoteRepoRootKeys =
["fe331502606802feac15e514d9b9ea83fee8b6ffef71335479a2e68d84adc6b0","1ea9ba32c526d1cc91ab5e5bd364ec5e9e8cb67179a471872f6e26f0ae773d42","2c6c3627bd6c982990239487f1abd02e08a02e6cf16edb105a8012d444d870c3","0a5c7ea47cd1b15f01f5f51a33adda7e655bc0f0b0615baa8e271f4c3351e21d","51f0161b906011b52c6613376b1ae937670da69322113a246a09f807c62f6921"],
remoteRepoKeyThreshold = 3, remoteRepoShouldTryHttps = True}
index-state(hackage.haskell.org) =
Searching for pkg-config in path.
Found pkg-config at /usr/bin/pkg-config
/usr/bin/pkg-config --version
/usr/bin/pkg-config is version 0.29
/usr/bin/pkg-config --variable pc_path pkg-config
Searching for pkg-config in path.
Found pkg-config at /usr/bin/pkg-config
/usr/bin/pkg-config --version
/usr/bin/pkg-config is version 0.29
/usr/bin/pkg-config --list-all
/usr/bin/pkg-config --modversion systemd formw menuw menu 'ncurses++' panelw panel ncurses tinfo tic ncursesw form 'ncurses++w' udev
Resolving dependencies...
targets: b
constraints: 
  any.base installed (non-upgradeable package)
  any.ghc-prim installed (non-upgradeable package)
  any.integer-gmp installed (non-upgradeable package)
  any.template-haskell installed (non-upgradeable package)
  setup.Cabal >=3.2 (minimum version of Cabal used by Setup.hs)
  setup.Cabal <3.4 (maximum version of Cabal used by Setup.hs)
  b.b ==1 (user target)
preferences: 
  b [TestStanzas,BenchStanzas]
strategy: PreferLatestForSelected
reorder goals: False
count conflicts: True
fine grained conflicts: True
minimize conflict set: False
independent goals: False
avoid reinstalls: False
shadow packages: False
strong flags: False
allow boot library installs: False
only constrained packages: OnlyConstrainedNone
max backjumps: 4000
[__0] trying: b-1 (user goal)
[__1] trying: base-4.14.0.0/installed-4.14.0.0 (dependency of b)
[__2] trying: rts-1.0/installedrts (dependency of base)
[__3] trying: integer-gmp-1.0.3.0/installed-1.0.3.0 (dependency of base)
[__4] trying: ghc-prim-0.6.1/installed-0.6.1 (dependency of base)
[__5] next goal: a (dependency of b)
[__5] trying: a-1/installed-7i1EAAn7OKY8zea1XCf99I
[__6] done
Elaborating the install plan...
File monitor 'source-hashes' changed: first run
Component graph for b-1: component lib:bar
CallStack (from HasCallStack):
  dieNoWrap, called at ./Distribution/Utils/LogProgress.hs:61:9 in Cabal-3.2.0.0-DWUJBcdbHvD7uNp9v26EzF:Distribution.Utils.LogProgress
Error:
    Dependency on unbuildable library 'foo' from a
    In the stanza 'library bar'
    In the inplace package 'b-1'

It's the same error!
Unfortunately due to the failure it isn't possible to extract the plan.json, but by looking at the output it would seem that either cabal-install or Cabal decided to depend on the main a library (7i1EAAn7OKY8zea1XCf99I is the id of the main library)

In fact, the same error happens when manually invoking Setup.hs configure without the --dependency flag or with an invalid one (either by changing a:foo to a, or by changing the ipid):

Error: Dependency on unbuildable library 'foo' from a

The above was on cabal 3.2. On master:

Resolving dependencies...
cabal: Could not resolve dependencies:
[__0] trying: b-1 (user goal)
[__1] next goal: a (dependency of b)
[__1] rejecting: a-1/installed-7i1EAAn7OKY8zea1XCf99I (does not contain
library 'foo', which is required by b)
[__1] fail (backjumping, conflict set: a, b)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: b, a

So even the solver tries to use the main library

Questions:

• why isn't cabal-install passing --dependency flags for installed packages (or passing wrong ones)?
• why isn't the basic Cabal solver solving the right library? Is it a bug that it doesn't or should downstream tools use --dependency?

[phadej]

It's sad that reproducing this needs messing with global-db.

Whatever this is, it does look like a bug.

Compare to running ./Setup configure on package with unsatisfiable constraints (i.e. no dependencies in package-db provided).

Configuring base64-bytestring-type-1.0.1...
Setup.hs: Encountered missing dependencies:
QuickCheck >=2.11.3 && <2.15,
aeson >=1.2.3.0 && <1.6,
base-compat >=0.9.3 && <0.12,
base64-bytestring >=1.0.0.1 && <1.3,
hashable >=1.2.6.1 && <1.4

So Distributions.Simple.Configure seems to proceed somewhere where invariants&assumptions do not hold.

[fgaz]

By the way that was on cabal 3.2, I updated the OP with the behavior of @grayjay's solver patch

[fgaz]

It's sad that reproducing this needs messing with global-db.

Yeah. That --package-db flag should really be implemented (#5773), so that it's easier to provide prebuilt packages to cabal from many sources

[fgaz]

@gbaz I think this is only a bug in Cabal's solver, not cabal-install's, since it can be reproduced with the Setup.hs interface too

edit: nevermind, i even wrote it in the op:

So even the [cabal-install] solver tries to use the main library