Resolving dependencies for wai repo on GHC 7.6.3 times out with Travis

[snoyberg]

We have regular build failures on the wai repo due to the build timing out, e.g. https://travis-ci.org/yesodweb/wai/jobs/132019472. I don't know if there is legitimately no build plan, or if the solver is simply having difficulty finding one. I've kicked off some dependency solving on one of my servers to try to determine if there's a workaround. Nonetheless, taking over 10 minutes to solve dependencies is surprising.

Reproduction:

• Tested with GHC 7.6.3 on Linux
• Tested with cabal-install 1.16, 1.22, and 1.24
• Check out the wai repo at commit c9427c887a40dbc35b6f4cca58f5096cd0a1b6e4 to the directory /home/travis/build/yesodweb/wai
• Run the following command:

cabal install --enable-tests --enable-benchmarks --force-reinstalls --ghc-options=-O0 --reorder-goals --max-backjumps=-1 /home/travis/build/yesodweb/wai/warp/ /home/travis/build/yesodweb/wai/wai-conduit/ /home/travis/build/yesodweb/wai/wai-websockets/ /home/travis/build/yesodweb/wai/wai/ /home/travis/build/yesodweb/wai/wai-handler-launch/ /home/travis/build/yesodweb/wai/auto-update/ /home/travis/build/yesodweb/wai/warp-tls/ /home/travis/build/yesodweb/wai/wai-handler-fastcgi/ /home/travis/build/yesodweb/wai/mime-types/ /home/travis/build/yesodweb/wai/wai-extra/ /home/travis/build/yesodweb/wai/wai-app-static/

Expected:

• Either dependency solving succeeds or fails, in a reasonable amount of time
• Very long period of inactivity

[23Skidoo]

/cc @kosmikus

[grayjay]

I started looking into this issue. There is no solution, because there is a conflict between warp's doctest test suite and its benchmarks with GHC 7.6.3.

The test requires doctest, which requires the package ghc. ghc cannot be reinstalled, and the installed version depends on the installed bytestring (0.10.0.2). The benchmark requires criterion >= 1. criterion-1.0.0.0 through 1.1.1.0 all require aeson >= 0.8, and aeson-0.8.0.0 through 0.11.2.0 require bytestring >= 0.10.4.0.

It would be nice to have cabal fail faster, especially when there is a conflict involving so few packages.

[snoyberg]

@grayjay Thank you, that information was very useful. For my immediate needs, I've disabled benchmark building for GHC 7.6.3 (yesodweb/wai@7127092), which works around this issue nicely, and we get testing of the wai repo with 7.6.3 again.

[phadej]

@grayjay is the problem because solver tries ghc quite late in the traversal?

[grayjay]

cabal tries ghc late because it waits to try warp:test. I ran the solver after removing the constraint on criterion so that it could finish. Here are the logs, with and without --reorder-goals: https://gist.github.com/grayjay/5e889f9a701705399f3a054fbc2ac312. I think it would be better if --reorder-goals always ranked goals with 0 or 1 valid choices, like warp:test, ahead of goals with more choices.

[kosmikus]

@grayjay Sorry, only now looking.

I think it would be better if --reorder-goals always ranked goals with 0 or 1 valid choices, like warp:test, ahead of goals with more choices.

Isn't this exactly what --reorder-goals is doing? Or are you saying that this strategy interferes with other heuristics? If you're already set up to test this, could you try with my or your "count-conflicts" branch?

[grayjay]

@kosmikus --reorder-goals prefers goals with fewer choices, but only after many choices have been removed by the heuristics phase. PSQ.prefer and preferByKeys remove the non-preferred goals:

cabal/cabal-install/Distribution/Solver/Modular/PSQ.hs

Line 136 in 9d0c0c0

prefer :: (a -> Bool) -> PSQ k a -> PSQ k a

.

I tried your count-conflicts branch, and it finished in about 8 minutes. The final conflict set was aeson, base, bytestring, criterion, directory, doctest, ghc, hashable, http2, psqueues, time, unix, wai-extra, warp, warp-3.2.7:test, warp-3.2.7:bench.

[kosmikus]

@grayjay Oh right. I had indeed discovered that in most of the cases where count-conflicts is still slow, the removal of choices in the heuristics phase is the culprit. I think we can/should fix this by switching to a scoring approach, as we've discussed elsewhere. Then we can more easily defer choices without completely removing them.

[snoyberg]

I'm unfamiliar with how the solver works internally and therefore have no
input, but just wanted to say thank you all for investing time in trying to
resolve this.

On Fri, May 27, 2016 at 10:25 AM, Andres Löh notifications@github.com
wrote:

@grayjay https://github.com/grayjay Oh right. I had indeed discovered
that in most of the cases where count-conflicts is still slow, the removal
of choices in the heuristics phase is the culprit. I think we can/should
fix this by switching to a scoring approach, as we've discussed elsewhere.
Then we can more easily defer choices without completely removing them.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#3453 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AADBB7SwJMelL5kWudhMj0uOEJn2Ydtyks5qFpxbgaJpZM4Ij_b1
.

[grayjay]

@kosmikus Yes, I think that would fix the issue. How will --reorder-goals interact with the goal ordering on your branch, or will it no longer be necessary?

@snoyberg No problem