Skip to content

ci(release): serialize release runs and pin Latest to the newest main tag - #56

Merged
automagik-genie merged 3 commits into
mainfrom
fix/release-race
Sep 26, 2026
Merged

automagik-genie merged 3 commits into
mainfrom
fix/release-race

Conversation

@automagik-genie

@automagik-genie automagik-genie commented Sep 26, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

One merge train (#50, #51, #54) pushed three merge commits to main within 14 seconds. Each push ran auto-release, which tagged it (v2.260926.3 on 2d6b6b6, .4 on 7d1a335, .5 on 6f908d4) and dispatched release.yml. The three release runs built in parallel, and GoReleaser marks every release it publishes as Latest (make_latest defaults to true). v2.260926.5 published at 20:57:42Z and v2.260926.4 at 20:57:43Z, so the Latest badge landed on v2.260926.4, a commit without #54.

Two races were involved:

  1. Tagging. Parallel auto-release runs each read the tag list before the others had pushed, so they could compute the same N or tag out of order.
  2. Latest. Whichever release run published last took Latest, whatever its tag.

Fix

File Change
.github/workflows/auto-release.yml concurrency: {group: auto-release, cancel-in-progress: false}. The job-level if on head_commit.message is gone; the job runs .github/scripts/next-release-tag.sh.
.github/scripts/next-release-tag.sh Skips a HEAD that is already tagged. Reads [skip release] and [skip ci] from every untagged first-parent commit since the last tag. It tags HEAD when any of them lacks a marker and skips only when all of them carry one. Calver N is computed as before, without the sed shellcheck flagged.
.github/workflows/release.yml Workflow concurrency: release-<tag>: runs for the same tag wait for each other. A new latest job runs after goreleaser in a global release-latest group (cancel-in-progress: false) and passes its own tag as RELEASE_TAG.
.github/scripts/set-latest-release.sh See the next section.
.goreleaser.yaml release.make_latest: "false", so publishing never moves Latest.
docs/RELEASING.md How tagging, releases and Latest interact. Replaced pending runs show as cancelled and lose nothing; a failed latest job is the signal to act on. How to check or repair Latest by hand.
internal/releasescripts/release_scripts_test.go Runs both scripts against throwaway git repos and a fake gh (it applies the script's --jq filters with real jq), so make ci covers these rules.

set-latest-release.sh

  • One release listing (gh api --paginate repos/…/releases). Any API error fails the job; it is never read as "no release". releases/latest returning 404 means there is no Latest yet; any other error fails the job.
  • It walks main's first-parent history from the tip and promotes the newest published release. Published means not a draft, not a prerelease, and with assets. The walk stops at the current Latest's commit, so Latest never moves backwards.
  • A newer tag on main whose release is missing, a draft or incomplete gets a ::warning:: naming it, and Latest falls back to the newest published release.
  • The job fails when its own tag is one of those unpublished newer tags after GoReleaser finished.

Asset completeness (review item 4)

GoReleaser v2 creates the release as a draft, uploads every artifact, and only then publishes it. This is internal/client/github.go, CreateRelease with the comment "Always start with a draft release while uploading artifacts. PublishRelease will undraft it", and internal/pipe/release/release.go, which calls PublishRelease after g.Wait(). Checked in the v2.12.7 source. v2.260926.5 matches: published_at 20:57:42Z comes after its last asset (20:57:41Z), with 8 assets.

So a non-draft GoReleaser release is complete, and a failed upload leaves a draft, which is never promoted. The script also refuses a published release with zero assets. I did not add a per-release expected asset count: it would duplicate the GoReleaser target matrix.

Why builds stay parallel

GitHub keeps only one pending run per concurrency group. A newer queued run replaces an older pending one, even with cancel-in-progress: false, and the replaced run shows as cancelled. With one global group around the whole release workflow, a train of three would silently lose the middle release. So only the Latest step is serialized globally. It derives its target from main and not from its own run, so a replaced pending latest job loses nothing. In auto-release, the replacing run reads every untagged commit, so an earlier push that wanted a release still gets one; the tag lands on the newer commit that contains it.

Simulation matrix (go test ./internal/releasescripts/, all PASS)

TestNextReleaseTag:

Case Result
no tags yet v2.260926.1
next number today (.1, .2 exist) v2.260926.3
first release of a new day v2.260926.1
HEAD already tagged skip
B wants a release, C [skip release] replaced B's pending run tags HEAD v2.260926.2
every untagged commit [skip release] / [skip ci] skip
[skip ci] only on a PR branch commit tags
[skip release] on the merge commit skip

TestSetLatestRelease (main has v2.260926.3/.4/.5 on first-parent merges):

Case rc Latest after Notes
incident: Latest .4, all published 0 .5 edit .5
already Latest 0 .5 no edit
no Latest yet (404) 0 .5 edit .5
newest (.5) missing, .4's run 0 .4 (from .3) ::warning::v2.260926.5 … missing
newest (.5) missing, .5's own run 1 .4 warning plus ::error::This run's tag v2.260926.5
newest build failed (draft), own run 1 .4 "its release is draft"
newest draft, other run, Latest .4 0 .4 "Latest stays v2.260926.4"
newest without assets 0 .4 "without-assets"
newest prerelease 0 .4 "prerelease"
release listing API error (HTTP 502) 1 unchanged ::error::Could not list the releases
Latest lookup API error 1 unchanged ::error::Could not read the current Latest
.6 missing, Latest .5, own .6 (never backwards) 1 .5 fallback stops at .5 and does not reach .4
older re-release (own .3), Latest .5 0 .5 no edit
untagged commits past .5 0 .5
newer-numbered tag off main (.9) 0 .5 .9 ignored

Revert checks

  • set-latest-release.sh from the first head of this PR (7d0d165), run with a fake that also answers its gh release view: 9 of 15 cases fail. The failures are both missing-release fallbacks, both API-error cases, both own-run failures, the draft-other-run warning, without-assets (it promoted a release with no assets) and never-backwards. The 6 cases that pass are behaviours the first version already had.
  • next-release-tag.sh judging only HEAD (the old job-level if): skip head replaces a pending release fails, and the other 7 pass.

Evidence

  • Latest was repaired by hand before this PR. gh api showed v2.260926.5 dereferences to 6f908d436e8fd9c16d122e352d2e029f9d6a5957 = commits/main. Then gh release edit v2.260926.5 --latest (rc 0).
  • shellcheck .github/scripts/*.sh: clean. actionlint v1.7.7: only the pre-existing custom runner-label notices; the SC2001 on the old calver sed is gone.
  • goreleaser check (v2.12.7): valid. The three archives.* deprecations are identical on origin/main.
  • make ci at 2c4baea (throwaway HOME, WK_* unset, own TMPDIR, GOFLAGS=-timeout=45m): rc 0, deadcode baseline check: OK, coverage total: 70.6% (min 70.0%).

The workflow changes take effect on the next push to main after merge, because release runs are dispatched with --ref main.

🤖 Generated with Claude Code

… tag

One merge train (#50, #51, #54) produced v2.260926.3, .4 and .5 within
seconds. Their release runs published in parallel, GoReleaser marked each
release Latest as it published, and .4 finished one second after .5, so
Latest landed on v2.260926.4 (7d1a335), which lacks #54.

- auto-release.yml: a `concurrency: auto-release` group
  (cancel-in-progress: false) runs tagging one push at a time, so a run
  reads the tags the previous run pushed; a commit that is already tagged
  is skipped.
- release.yml: runs for the same tag share a group and wait for each
  other. A new `latest` job, in a global `release-latest` group
  (cancel-in-progress: false), runs after GoReleaser and marks Latest only
  on the release whose tag is on the newest tagged main commit, and only
  once that release is published (.github/scripts/set-latest-release.sh).
- .goreleaser.yaml: make_latest "false", so publishing never moves Latest.
- docs/RELEASING.md: how tagging, releases and Latest interact, and how to
  check or repair Latest by hand.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Sep 26, 2026 •

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 53aa9a85-9637-41f5-b703-80411d1bfde0


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

namastex888 and others added 2 commits September 26, 2026 22:21
Review follow-ups for the release race fix.

- auto-release: the job-level `if` read only head_commit.message. When a
  `[skip release]` run replaced the pending run of an earlier push that
  wanted a release, neither commit was tagged. The skip markers are now
  read inside the job over every untagged first-parent commit since the
  last tag (.github/scripts/next-release-tag.sh): HEAD is tagged when any
  of them lacks a marker, and skipped only when all of them carry one.
- set-latest-release.sh: lists releases once and fails on any API error
  instead of reading it as "missing"; a 404 from releases/latest means no
  Latest yet. It walks main from the tip and promotes the newest published
  release (non-draft, non-prerelease, with assets), never past the current
  Latest. A newer unpublished tag gets a ::warning:: naming it, and the job
  fails when its own tag (RELEASE_TAG) is that unpublished tag.
- GoReleaser creates releases as drafts and publishes only after every
  upload (v2 internal/client/github.go CreateRelease/PublishRelease), so a
  non-draft release is complete; a failed upload leaves a draft.
- docs/RELEASING.md: replaced pending runs show as cancelled and lose
  nothing; a failed `latest` job is the signal.
- internal/releasescripts: drives both scripts against throwaway git repos
  and a fake gh, so `make ci` covers the tagging and Latest rules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@automagik-genie
automagik-genie merged commit 6c30237 into main Sep 26, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants