Shall we use head.hackage in CI for the newest GHC (or all GHCs?)

[Mikolaj]

We discussed

#9610 (comment)

on a cabal fortnightly chat and during the discussion @ulysses4ever proposed a somehow related idea to move our CI (at least for the newest GHC) to https://ghc.gitlab.haskell.org/head.hackage/. I think this should have a higher priority than the original proposal to fix --index-state in our github CI. Let's discuss (and volunteer for subtasks). E.g.,

1. Whether to fix the index state of head.hackage or just follow the HEAD of head.hackage, which is quite a bit curated and fixed often.
2. Whether to use head.hackage for other GHCs and, if so, exclusively or in addition to newest Hackage and/or fixed index state.
3. Maybe add to the mixture --dry-run or some other policy, e.g., to run short tests in many configurations and long ones only in one (but we don't only worry about CI time, but also information overload, especially for new contributors that have to read CI results).
4. The same on our release CI that is located on Haskell gitlab instance.

In addition, let's discuss the original proposal, which included the --index-state proposal for the whole github CI and more. Here's are some remarks about the proposal from the cabal fortnightly chat:

Agree with Bryan's points from the discussion that fixing the index state is useful for a more robust CI.

There is no argument against it, but it is not high priority. Accomodating head.hackage would be a higher priority. Fixing the index state wouldn't save you from dealing with failures due to Hackage updates, it just puts you on a more predictable schedule of when you deal with those failures (points where you update the state).

Unless the failures are already fixed by the point we update our fixed index. But by then we may be inundated by user reports that we can't reproduce in our CI, which is yet another way in which we can't escape the failures some of the time, but they don't disrupt our regular PR merging process at least. There is also the related issue of isolating ourselves from GHA (via containers or custom ghcup invocations). Similar considerations apply. I estimate each has bitten us ~5 times in the last couple of years. That's not many times, but that kind of firefighting tends to burn out maintainers (however, nothing fully isolates us from this, because stack and cabal are the go-to bug trackers for affected users).

CC: @chreekat @philderbeast

[geekosaur]

One point that occurred to me after the call: periodically we bump bounds, and those PRs need to be run without an index-state if we do otherwise freeze it for CI.

[chreekat]

Without putting a lot of thought into it, my immediate feeling is that head.hackage is not stable enough to be used as the basis for CI. After putting a few more seconds of thought into it, I do think it is possible, if done with care.

Some of the questions in the OP don't make sense to me, so I wonder if we lack a clear shared understanding of what head.hackage is. For instance, I understand that head.hackage is only suitable for use with GHC HEAD, i.e. the tip of GHC's master branch. Is that what this ticket is about?

[Mikolaj]

@chreekat: thank you for your thoughts. I think you are right head.hackage is likely to break with old GHCs, even though it often works, I think (it's an overlay, so old packages are still visible, right?). In any case, this was intended to be used for testing the GHC version that is not yet released, but that a cabal major release needs to sync with despite it not being released. Also, it would let us test against the versions of package prepared for the newest GHC, which we can't test from normal Hackage. It would also often catch problems with newest versions of package on Hackage not forced by a GHC release process, because the build plans for that pipeline will normally not exclude any packages as too new.

That CI would be the high-breakage part of our CI, maybe even run overnight and not with every PR. When it's in place, we could make the rest of CI conservative, e.g., sharing the same index state pin as the release CI. Does this clarify things?

[chreekat]

In any case, this was intended to be used for testing the GHC version that is not yet released, but that a cabal major release needs to sync with despite it not being released

Ok! head.hackage is not only good but necessary for this use case.

But I would not recommend trying this use case until https://gitlab.haskell.org/ghc/ghc/-/issues/24000 is solved. I mean, you can try, but I don't know if the cost-benefit tradeoff would be worth it.

[chreekat]

Here's my GHC Nightly status dashboard: https://grafana.gitlab.haskell.org/d/ab109e66-a8a1-4ae9-b976-40e2dfe281ab/availability-of-ghc-nightlies-via-ghcup?orgId=2&refresh=1d

[philderbeast]

$ curl https://ghc.gitlab.haskell.org/head.hackage/cabal.project >> cabal.project.local
$ cabal update

I followed the above head.hackage instructions with master. The default cabal.project builds but not at first;

$ cabal --version
cabal-install version 3.11.0.0
compiled using version 3.11.0.0 of the Cabal library

$ git rev-parse HEAD
46e822152c9b3f38044870115d7a198b3a509d66

$ cabal build all --enable-tests --enable-benchmarks --dry-run
Warning: this is a debug build of cabal-install with assertions enabled.
Resolving dependencies...
Error: [Cabal-7107]
Could not resolve dependencies:
[__0] next goal: Cabal (user goal)
[__0] rejecting: Cabal-3.11.0.0
      (cabal.project.local requires ==2.4.1.0 || ==3.0.2.0 || ==3.2.1.0)
[__0] rejecting: Cabal-3.10.2.1, Cabal-3.10.2.0/installed-fd40, Cabal-3.10.2.0, Cabal-3.10.1.0, Cabal-3.8.1.0, Cabal-3.6.3.0, Cabal-3.6.2.0, Cabal-3.6.1.0, Cabal-3.6.0.0, Cabal-3.4.1.0, Cabal-3.4.0.0, Cabal-3.2.1.0, Cabal-3.2.0.0, Cabal-3.0.2.0, Cabal-3.0.1.0, Cabal-3.0.0.0, Cabal-2.4.1.0, Cabal-2.4.0.1, Cabal-2.4.0.0, Cabal-2.2.0.1, Cabal-2.2.0.0, Cabal-2.0.1.1, Cabal-2.0.1.0, Cabal-2.0.0.2, Cabal-1.24.2.0, Cabal-1.24.0.0, Cabal-1.22.8.0, Cabal-1.22.7.0, Cabal-1.22.6.0, Cabal-1.22.5.0, Cabal-1.22.4.0, Cabal-1.22.3.0, Cabal-1.22.2.0, Cabal-1.22.1.1, Cabal-1.22.1.0, Cabal-1.22.0.0, Cabal-1.20.0.4, Cabal-1.20.0.3, Cabal-1.20.0.2, Cabal-1.20.0.1, Cabal-1.20.0.0, Cabal-1.18.1.7, Cabal-1.18.1.6, Cabal-1.18.1.5, Cabal-1.18.1.4, Cabal-1.18.1.3, Cabal-1.18.1.2, Cabal-1.18.1.1, Cabal-1.18.1, Cabal-1.18.0, Cabal-1.16.0.3, Cabal-1.16.0.2, Cabal-1.16.0.1, Cabal-1.16.0, Cabal-1.14.0, Cabal-1.12.0, Cabal-1.10.2.0, Cabal-1.10.1.0, Cabal-1.10.0.0, Cabal-1.8.0.6, Cabal-1.8.0.4, Cabal-1.8.0.2, Cabal-1.6.0.3, Cabal-1.6.0.2, Cabal-1.6.0.1, Cabal-1.4.0.2, Cabal-1.4.0.1, Cabal-1.4.0.0, Cabal-1.2.4.0, Cabal-1.2.3.0, Cabal-1.2.2.0, Cabal-1.2.1, Cabal-1.1.6, Cabal-1.24.1.0 (constraint from user target requires ==3.11.0.0)
[__0] fail (backjumping, conflict set: Cabal)
After searching the rest of the dependency tree exhaustively, these were the goals I've had most trouble fulfilling: Cabal

I can fix this by commenting out the constraint conflict on Cabal in cabal.project.local;

$ git diff
diff --git a/cabal.project.local b/cabal.project.local
index 576224433..9b6fb23cb 100644
--- a/cabal.project.local
+++ b/cabal.project.local
@@ -31,7 +31,7 @@ constraints:
     template-haskell installed
 
 constraints:
-    Cabal ==2.4.1.0 || ==3.0.2.0 || ==3.2.1.0,
+ -- Cabal ==2.4.1.0 || ==3.0.2.0 || ==3.2.1.0,
     Cabal-syntax ==3.8.1.0,
     FPretty ==1.1,
     JuicyPixels ==3.3.8,

Try again and there's another conflict for Cabal-syntax that I can fix the same way;

$ cabal build all --enable-tests --enable-benchmarks --dry-run
Warning: this is a debug build of cabal-install with assertions enabled.
Resolving dependencies...
Error: [Cabal-7107]
Could not resolve dependencies:
[__0] next goal: Cabal-syntax (user goal)
[__0] rejecting: Cabal-syntax-3.11.0.0
      (constraint from cabal.project.local requires ==3.8.1.0)
[__0] rejecting: Cabal-syntax-3.10.2.0/installed-8eb4, Cabal-syntax-3.10.2.0, Cabal-syntax-3.10.1.0, Cabal-syntax-3.8.1.0, Cabal-syntax-3.6.0.0 (constraint from user target requires ==3.11.0.0)
[__0] fail (backjumping, conflict set: Cabal-syntax)
After searching the rest of the dependency tree exhaustively, these were the goals I've had most trouble fulfilling: Cabal-syntax

$ git diff
diff --git a/cabal.project.local b/cabal.project.local
index 576224433..c92471375 100644
--- a/cabal.project.local
+++ b/cabal.project.local
@@ -31,8 +31,8 @@ constraints:
     template-haskell installed
 
 constraints:
-    Cabal ==2.4.1.0 || ==3.0.2.0 || ==3.2.1.0,
-    Cabal-syntax ==3.8.1.0,
+ -- Cabal ==2.4.1.0 || ==3.0.2.0 || ==3.2.1.0,
+ -- Cabal-syntax ==3.8.1.0,
     FPretty ==1.1,
     JuicyPixels ==3.3.8,
     ansi-pretty ==0.1.2.2,

Now cabal build ... works.

[philderbeast]

Who maintains https://ghc.gitlab.haskell.org/head.hackage/? It would be nice to have 4 space indenting in the project to allow more room for -- commenting inplace.

[philderbeast]

• Why does https://ghc.gitlab.haskell.org/head.hackage/cabal.project have an allow-newer exception for Cabal but also equality version constraints on Cabal?

• Why is there an allow-newer exception for Cabal but not for Cabal-syntax? They're both boot libraries aren't they, looking at the ghc-9.8.1 included libraries?

[andreabedini]

I'm 👎 on adding head.hackage to the CI. I admit I have been out of the loop and missed some conversations but I fail to see what problems using head.hackage will solve.

[geekosaur]

The main one is that, if you are making a Cabal release intended for an unreleased ghc, you generally need to use head.hackage to get compatible dependencies.