'cabal install curl' fails when given the --allow-newer flag

[RyanGlScott]

A bizarre bug, but luckily a pretty reproducible one. cabal install curl works fine, but adding the --allow-newer flag makes the installation go kaput. I've reproduced this with both cabal-install-1.22.8.0/Cabal-1.22.5.0/GHC 7.10.3 and cabal-install-1.24.0.0/Cabal-1.24.0.0/GHC 8.0.1 on two different 64-bit Linux machines. Here's the gist of the error with Cabal-1.24.0.0:

$ /opt/cabal/1.24/bin/cabal install curl --allow-newer -j1
Resolving dependencies...
cabal: Entering directory '/tmp/cabal-tmp-18165/curl-1.3.8'
Configuring curl-1.3.8...
checking for gcc... /usr/bin/gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables... 
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether /usr/bin/gcc accepts -g... yes
checking for /usr/bin/gcc option to accept ANSI C... none needed
checking how to run the C preprocessor... /usr/bin/gcc -E
configure: creating ./config.status
config.status: creating curl.buildinfo
Building curl-1.3.8...
Preprocessing library curl-1.3.8...
[1 of 8] Compiling Network.Curl.Debug ( Network/Curl/Debug.hs, dist/build/Network/Curl/Debug.o )
[2 of 8] Compiling Network.Curl.Code ( Network/Curl/Code.hs, dist/build/Network/Curl/Code.o )
[3 of 8] Compiling Network.Curl.Types ( Network/Curl/Types.hs, dist/build/Network/Curl/Types.o )

Network/Curl/Types.hs:32:1: error:
    Failed to load interface for ‘Data.IntMap’
    It is a member of the hidden package ‘containers-0.5.7.1’.
    Perhaps you need to add ‘containers’ to the build-depends in your .cabal file.
    Use -v to see a list of the files searched for.
cabal: Leaving directory '/tmp/cabal-tmp-18165/curl-1.3.8'
Failed to install curl-1.3.8
cabal: Error: some packages failed to install:
curl-1.3.8 failed during the building phase. The exception was:
ExitFailure 1

Here are the full -v3 build logs for Cabal-1.22 and Cabal-1.24.

[23Skidoo]

Thanks for the bug report, I'll try to look into this issue today.

[hvr]

without having looked closer, I suspect this is exactly the kind of reason why an unscoped --allow-newer is a very bad idea as it interacts badly with automatic cabal flags during solving, if the flag choice relies on upper bounds being honored.

PS: See #2756 & #3171 for the scoped-allow-newer feature

[kosmikus]

@hvr is right:

$ cabal install curl --dry-run -v
[...]
Resolving dependencies...
In order, the following would be installed:
curl-1.3.8 +new-base (new package)

$ cabal install curl --dry-run --allow-newer -v
[...]
Resolving dependencies...
In order, the following would be installed:
curl-1.3.8 (new package)

From curl.cabal:

flag new-base
  Description: Build with new smaller base library
  Default: False
[...]
library
[...]
  if flag(new-base)
    Build-depends: base >= 3 && < 5, containers
  else
    Build-depends: base < 3

Flag default for new-base is set to False, and with global allow-newer, the base < 3 constraint is accepted.

[hvr]

Btw, here's a similar pathological xxx.cabal file which makes this effect observable as well:

name:                xxx
version:             0
build-type:          Simple
cabal-version:       >=1.10

flag f1
  manual: False
  default: True

library
  default-language: Haskell2010
  if flag(f1)
     build-depends: base<0.0
     hs-source-dirs: non-existent-broken-folder

$ cabal configure 
Resolving dependencies...
Configuring xxx-0...

vs.

$ cabal configure --allow-newer
Resolving dependencies...
Configuring xxx-0...
Warning: 'hs-source-dirs: non-existent-broken-folder' directory does not exist.

[23Skidoo]

So the right way to fix this is to make the new-base flag True by default in curl.cabal. We should also advise the users to prefer the --allow-newer=foo,bar,baz form and scoping instead of relaxing all upper bounds unconditionally.

It would be also nice if such surprising results were accompanied with a warning, but it's pretty hard to implement.

[hvr]

@23Skidoo tbh, I don't think there is any proper fix to this. Changing the default setting of the flag is just a workaround that happens to work now, but may very well fail to work in the future, unless we make it a guaranteed behaviour of the solver (which I wouldn't). IMO, --allow-newer is just a fragile hack that may or may not work as intended. And there's other cases which don't involve cabal flags where --allow-newer result in worse solutions than without --allow-newer

[grayjay]

Should cabal only ignore the highest upper bound for each dependency? Then it would only remove 5 as an upper bound in the first example:

  if flag(new-base)
    Build-depends: base >= 3, containers
  else
    Build-depends: base < 3

I think that bounds that are lower than the maximum within a package are more likely to have been added because of known build failures, so there is less reason to remove them.

[23Skidoo]

@grayjay Your suggestion makes sense, but I'm not completely sure that this change won't make --alow-newer break in some other way. We'll need to evaluate it somehow before applying.

[hvr]

@grayjay but then the --allow-newer transformation wouldn't be idempotent anymore which seems even more fragile than the current scheme

[grayjay]

Yeah, I'm not sure it's an improvement. It could give worse results in some cases, like when an author forgets to raise the upper bound on a dependency in one component, and a user wants to lift that bound specifically.