Migrate release CI back to github #9437

hasufell · 2023-11-13T09:43:50Z

No description provided.

hasufell · 2023-11-13T15:42:45Z

Example pipeline: https://github.com/haskell/cabal/actions/runs/6851573157/job/18629453051?pr=9437
Example draft release: https://github.com/haskell/cabal/releases/tag/untagged-5a72917f7c0a5a2480ec
Example cirrus CI pipeline: https://cirrus-ci.com/task/6646855723909120?logs=main#L11

Also includes:

script to download artifacts
script to create yaml snippet for ghcup

hasufell · 2023-11-13T15:55:01Z

Workflow:

push tag cabal-install-v4.0.0.0
wait for github and cirrus builds to finish
run scripts/release/download-gh-artifacts.sh cabal-install-v4.0.0.0 <your-gpg-email>
- upload the contents of the folder gh-release-artifacts/cabal-4.0.0.0 to https://downloads.haskell.org/~cabal/cabal-install-4.0.0.0/
run scripts/release/create-yaml-snippet.sh cabal-install-v4.0.0.0
- based on the output create a ghcup-metadata PR

Kleidukos

From my perspective this is quite appealing and I would like to give it a try for the next release.

chreekat · 2023-11-15T09:27:07Z

Time comparison:

Gitlab CI: 98m 15s
    https://gitlab.haskell.org/haskell/cabal/-/pipelines/86388
GitHub CI: 32m 35s
    https://github.com/haskell/cabal/actions/runs/6851573157

Ironically, the GitLab pipeline took forever because the Darwin job had to queue
for an hour for an open spot. The GitHub job runs on the same server and
therefore competes for the same resource. It just gets to jump the queue since
the two job runners don't know about each other.

hasufell · 2023-11-15T09:42:21Z

I do think the GitLab stuff should be kept as a backup as part of good change management, however, so I will request that change.

That would mean we just increased complexity, which defeats the purpose of this PR.

If anything goes extremely wrong during the next release, you can restore the gitlab code from the git history.

chreekat · 2023-11-15T10:09:17Z

That would mean we just increased complexity, which defeats the purpose of this PR.

If it wasn't clear, I meant to keep it around for the next release and then dropping it. I hope that doesn't defeat the purpose. This change is bigger than just reimplementing some code ,and keeping two systems running in parallel is good practice. And basically free in this case.

Mikolaj · 2023-11-15T10:19:10Z

keep it around for the next release and then dropping it

Regarding this one point, I'd make it two releases and I'd keep the gitlab CI running all the time to ensure we spot problems in it ASAP, just as we do today.

While I'm talking, @hasufell, thank you for the PR. I remember the big appeal of your proposal long ago, now materialised, was that we'd test what we release. Does this PR bring us closer to the point where we test daily the same binaries that will be released? In particular, are our normal GHA tests run on these artifacts, and when not applicable, on the exact commits from which the artifacts are created? What's the roadmap to unify CI and release CI, to a reasonable extent?

hasufell · 2023-11-16T01:35:28Z

That would mean we just increased complexity, which defeats the purpose of this PR.

If it wasn't clear, I meant to keep it around for the next release and then dropping it. I hope that doesn't defeat the purpose. This change is bigger than just reimplementing some code ,and keeping two systems running in parallel is good practice.

I don't agree with this at all. There's no indication this is necessary. In HLS we also didn't need it and it has a much more complex CI.

chreekat · 2023-11-16T08:20:27Z

That would mean we just increased complexity, which defeats the purpose of this PR.

If it wasn't clear, I meant to keep it around for the next release and then dropping it. I hope that doesn't defeat the purpose. This change is bigger than just reimplementing some code ,and keeping two systems running in parallel is good practice.

I don't agree with this at all. There's no indication this is necessary. In HLS we also didn't need it and it has a much more complex CI.

To avoid a deadlock I've created https://github.com/chreekat/cabal/tree/b/add-github-release-ci, which is an edited version of this branch that keeps GitLab CI. I hope it won't be necessary.

hasufell · 2023-11-16T09:16:55Z

I can split the PR in two commits, so that Gitlab removal is easier to revert in case it's necessary.

But I won't subscribe to "let's keep code in master, just in case". That's bad engineering practice.

Mikolaj · 2023-11-16T11:28:41Z

But I won't subscribe to "let's keep code in master, just in case". That's bad engineering practice.

I don't think anybody is proposing that. Gitlab CI would be running continuously so that we can spot any problems and fix them ASAP and during a release, if needed, the release manager would be able to compare the results of gitlab and github, test behaviour of the artifacts from both systems, etc.

hasufell · 2023-11-16T11:30:59Z

I have decided to build bindists myself for the next cabal release, so cabal's CI isn't relevant for GHCup directly anymore.

You can use this PR and the contained patch as you please.

andreabedini · 2023-11-17T01:28:34Z

I have reopened this to prevent us losing valuable work.

andreabedini · 2023-11-18T04:02:59Z

And thank you @hasufell for working on this!

andreabedini

Just leaving some comments as I had the time to read this properly. It looks quite comprehensive! good stuff.

PS: @hasufell you have been clear that you don't intend to continue working on this so my comments are not to be taken as request for more work from you :-) They more like notes to my future self.

andreabedini · 2023-12-01T16:51:25Z

.github/scripts/build.sh

+
+# https://github.com/haskell/cabal/issues/7313#issuecomment-811851884
+if [ "$(getconf LONG_BIT)" == "32" ] || [ "${DISTRO}" == "CentOS" ] ; then
+    echo 'constraints: lukko -ofd-locking' >> cabal.project.release.local


I believe this should just be included in cabal.project.release under a if arch(I386) conditional. I am not sure if there is something specific to CentOS, it does not seem to be mentioned in the thread above

andreabedini · 2023-12-01T16:57:06Z

.github/scripts/build.sh

+args=(
+    -w "ghc-$GHC_VERSION"
+    --disable-profiling
+    --enable-executable-stripping


other things that should be in cabal.project.release

andreabedini · 2023-12-01T16:58:28Z

.cirrus.yml

+  timeout_in: 120m
+  only_if: $CIRRUS_TAG != ''
+  env:
+    ADD_CABAL_ARGS: "--enable-split-sections"


this can be implemented in cabal.project.release under a conditional if os(FreeBSD)

Why would this be conditional on platform? AFAIK it's only Windows that doesn't support it, and it's a win everywhere else. (Macs ignore the option since they always do dead code stripping, although GHC needed a lot of hackery to make use of it safely, but that's not our problem.)

andreabedini · 2023-12-01T17:03:55Z

.github/workflows/release.yaml

+        uses: actions/checkout@v3
+
+      - if: matrix.ARCH == 'ARM'
+        uses: docker://hasufell/arm32v7-ubuntu-haskell:focal


Are there officially maintaned images we can use?

I would expect that there are official Ubuntu images, but they don't have any Haskell stuff preinstalled and if they did it'd be Ubuntu's ancient versions.

The images are autorebuilt: https://github.com/haskell/ghcup-hs/actions/workflows/docker.yaml

However, you should be using docker://hasufell/arm32v7-debian-haskell:10.

See my latest action: https://github.com/stable-haskell/cabal/blob/cabal-install-v3.12.1.0/.github/workflows/release.yaml

andreabedini · 2023-12-01T17:04:03Z

.github/workflows/release.yaml

+          ARTIFACT: ${{ matrix.ARTIFACT }}
+
+      - if: matrix.ARCH == 'ARM64'
+        uses: docker://hasufell/arm64v8-ubuntu-haskell:focal


same as above

andreabedini · 2023-12-01T17:08:37Z

.github/workflows/release.yaml

+
+      - name: git config
+        run: |
+          git config --global --get-all safe.directory | grep '^\*$' || git config --global --add safe.directory "*"


in the current ci we just do

git config --global protocol.file.allow always

and we have a comment point to https://bugs.launchpad.net/ubuntu/+source/git/+bug/1993586 (cabal PR #8546)

geekosaur · 2024-06-25T03:24:11Z

I'd be willing to take this on after we get 3.12.1.0 out the door.

andreabedini · 2024-06-25T04:18:37Z

I'd be willing to take this on after we get 3.12.1.0 out the door.

Happy to help too.

geekosaur · 2024-09-18T07:12:13Z

I have temporarily reinstated the Gitlab CI stuff until we come to an agreement about it. (I'm on the side that it's always possible to arrange things so it can be reverted back in if we need it. I mean, that's what git is for, isn't it?)

(And part of the reason I did that is so I can take it out in a distinct commit that can easily be reverted.)

geekosaur · 2024-09-19T08:55:09Z

Manually squashed, added Authored-by: to keep original attribution.

geekosaur · 2024-09-19T08:57:26Z

Gitlab CI removed again, revert a12c6be to bring it back if needed.

@hasufell

Authored-by: Julian Ospald (@hasufell)

Revert this commit at need.

hasufell marked this pull request as draft November 13, 2023 09:43

hasufell force-pushed the purge-gitlab branch 16 times, most recently from 001d0e1 to bb01a96 Compare November 13, 2023 14:57

hasufell force-pushed the purge-gitlab branch 2 times, most recently from 5c37718 to 34a7c9c Compare November 13, 2023 15:45

hasufell marked this pull request as ready for review November 13, 2023 15:45

hasufell mentioned this pull request Nov 13, 2023

[Rel-Eng]: Adds script to automatically generate GHCup metadata for Cabal releases #9411

Merged

3 tasks

hasufell force-pushed the purge-gitlab branch 2 times, most recently from 3187a1a to 8781d59 Compare November 13, 2023 15:53

hasufell requested review from andreabedini and Kleidukos November 13, 2023 15:56

hasufell force-pushed the purge-gitlab branch 2 times, most recently from 81397df to 0800602 Compare November 13, 2023 16:09

Kleidukos previously approved these changes Nov 13, 2023

View reviewed changes

hasufell closed this Nov 16, 2023

andreabedini reopened this Nov 17, 2023

wz1000 mentioned this pull request Nov 28, 2023

HLS 2.5.0.0 haskell/ghcup-metadata#159

Merged

andreabedini reviewed Dec 1, 2023

View reviewed changes

hasufell mentioned this pull request Mar 21, 2024

Formalising our stance regarding binary distribution of cabal-install #9827

Open

geekosaur force-pushed the purge-gitlab branch from fbf1f85 to 8c78a11 Compare September 18, 2024 07:11

geekosaur force-pushed the purge-gitlab branch 2 times, most recently from 90334dd to 3f2515c Compare September 19, 2024 08:54

geekosaur added continuous-integration blocked: decision-needed labels Sep 20, 2024

geekosaur added 2 commits September 20, 2024 18:06

Migrate release CI back to GitHub

82001f8

Authored-by: Julian Ospald (@hasufell)

Remove GitLab CI

a12c6be

Revert this commit at need.

geekosaur force-pushed the purge-gitlab branch from f232a4a to a12c6be Compare September 20, 2024 22:07

geekosaur mentioned this pull request Sep 21, 2024

test on Tier 2 platforms and make a prerelease for them #10372

Draft

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate release CI back to github #9437

Migrate release CI back to github #9437

hasufell commented Nov 13, 2023

hasufell commented Nov 13, 2023

hasufell commented Nov 13, 2023

Kleidukos left a comment

chreekat commented Nov 15, 2023 •

edited

Loading

hasufell commented Nov 15, 2023

chreekat commented Nov 15, 2023

Mikolaj commented Nov 15, 2023 •

edited

Loading

hasufell commented Nov 16, 2023

chreekat commented Nov 16, 2023

hasufell commented Nov 16, 2023

Mikolaj commented Nov 16, 2023

hasufell commented Nov 16, 2023

andreabedini commented Nov 17, 2023

andreabedini commented Nov 18, 2023

andreabedini left a comment •

edited

Loading

andreabedini Dec 1, 2023

andreabedini Dec 1, 2023

andreabedini Dec 1, 2023

geekosaur Sep 20, 2024

andreabedini Dec 1, 2023

geekosaur Sep 19, 2024

hasufell Sep 19, 2024 •

edited

Loading

andreabedini Dec 1, 2023

andreabedini Dec 1, 2023

geekosaur commented Jun 25, 2024

andreabedini commented Jun 25, 2024

geekosaur commented Sep 18, 2024 •

edited

Loading

geekosaur commented Sep 19, 2024

geekosaur commented Sep 19, 2024 •

edited

Loading

Migrate release CI back to github #9437

Are you sure you want to change the base?

Migrate release CI back to github #9437

Conversation

hasufell commented Nov 13, 2023

hasufell commented Nov 13, 2023

hasufell commented Nov 13, 2023

Kleidukos left a comment

Choose a reason for hiding this comment

chreekat commented Nov 15, 2023 • edited Loading

hasufell commented Nov 15, 2023

chreekat commented Nov 15, 2023

Mikolaj commented Nov 15, 2023 • edited Loading

hasufell commented Nov 16, 2023

chreekat commented Nov 16, 2023

hasufell commented Nov 16, 2023

Mikolaj commented Nov 16, 2023

hasufell commented Nov 16, 2023

andreabedini commented Nov 17, 2023

andreabedini commented Nov 18, 2023

andreabedini left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hasufell Sep 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

geekosaur commented Jun 25, 2024

andreabedini commented Jun 25, 2024

geekosaur commented Sep 18, 2024 • edited Loading

geekosaur commented Sep 19, 2024

geekosaur commented Sep 19, 2024 • edited Loading

chreekat commented Nov 15, 2023 •

edited

Loading

Mikolaj commented Nov 15, 2023 •

edited

Loading

andreabedini left a comment •

edited

Loading

hasufell Sep 19, 2024 •

edited

Loading

geekosaur commented Sep 18, 2024 •

edited

Loading

geekosaur commented Sep 19, 2024 •

edited

Loading