ci: TEM-9 release workflow

[Chmokachka]

Summary

Implements automated, semver-based releases driven by Conventional Commits
(TEM-9) and documents the process (TEM-10).

Previously the version lived in versions.hcl and was bumped by hand. This PR
makes the version derived from git tags + Conventional Commits: PRs build
release-candidate images, and a squash-merge to main builds the final images,
creates the vX.Y.Z tag, and publishes a GitHub Release — with no manual
version editing.

A single orchestrator workflow computes the version once, fans out to the
per-family build workflows (reused via workflow_call), waits for all of them,
and only then cuts the release — so builds and the release can't disagree on the
version or race each other.

How it works

• One orchestrator (release.yml, "Build and Release") owns all triggers
  (pull_request, push: main, workflow_dispatch), computes the version once
  in a version job, and passes it into every build.
• Version source: the latest vX.Y.Z tag reachable from HEAD + the
  Conventional Commit type.
  • feat: → minor, fix:/perf: → patch, !/BREAKING CHANGE → major,
    everything else (ci:/chore:/docs:…) → no release.
  • We squash-merge, so on PRs the bump is derived from the PR title (which
    becomes the commit on main); on main it's derived from the commits.
• Per-family build workflows are reusable (base.yml, nvidia.yml,
  rocm.yml via on: workflow_call) — they no longer compute the version or
  own triggers; the orchestrator calls them with the computed version/suffix.
• On a PR: builds RC images X.Y.Z-rc.<PR#> for the affected families only
  (a changes job gates which reusable workflows are called).
• On merge to main (releasable): builds all families with X.Y.Z,
  then the release job tags vX.Y.Z + cuts a GitHub Release only after every
  family built and smoke-tested successfully.
• Manual workflow_dispatch: builds -dev images, never releases.
• No release race: the orchestrator serializes on push: main
  (concurrency with a constant group, cancel-in-progress: false), so two
  merges landing close together can't compute the same version — the 2nd waits
  for the 1st to tag.
• Dependency chain base → pytorch → pytorch-cluster: -cluster builds
  FROM the published runpod/pytorch, so it runs after build-pytorch in
  the same run. On a release it layers on the freshly-built X.Y.Z pytorch; on a
  PR that changed pytorch it layers on that PR's RC pytorch (so smoke tests
  reflect the real post-merge image); a cluster-only PR layers on the last
  released pytorch (no pointless full-matrix pytorch rebuild).

Changes

• New .github/actions/compute-version — computes version/suffix and the
  should-build / should-release flags from git + Conventional Commits.
  Selects the base tag as the latest release reachable from HEAD (so
  re-running an older commit doesn't pick up a newer, unrelated tag). Also
  exposes a base-version output (last released version, no bump) so a cluster
  build can layer onto an already-published pytorch when this run doesn't
  rebuild pytorch.
• release.yml is now the orchestrator — owns triggers + concurrency,
  runs the single version job and a changes job (per-family PR gating),
  calls the reusable build workflows, and gates the tag + GitHub Release on
  push: main + should-release + all families succeeding.
• base.yml / nvidia.yml / rocm.yml → reusable workflows
  (on: workflow_call, inputs version/suffix/base_version,
  secrets: inherit). Removed their individual version jobs, triggers and
  concurrency; RELEASE_VERSION/RELEASE_SUFFIX now come from inputs. Internal
  per-family changed-files gating (pytorch/autoresearch/cluster) is unchanged.
• pytorch-cluster release — the -cluster family lives inside base.yml
  as build-cluster + test-cluster after build-pytorch (base → pytorch →
  cluster in one deterministic pipeline). pytorch-cluster/docker-bake.hcl's
  BASE_IMAGE uses a dedicated PYTORCH_BASE_VERSION variable (the pytorch it
  builds FROM), kept separate from the cluster's own tag so PR builds don't try
  to layer onto a not-yet-published pytorch version.
• PR path filters use directory globs (official-templates/**,
  container-template/**) so no build-relevant input (e.g. requirements.txt,
  pre_start.sh, start.sh) is missed. Pipeline-only (ci:) changes to the
  workflow/action files intentionally don't trigger a build.
• softprops/action-gh-release pinned to a commit SHA (like the other
  actions).
• docker-setup — dropped the old -dev-<branch> suffix logic (moved into
  compute-version).
• versions.hcl — RELEASE_VERSION is now a bake variable (CI-overridable;
  the default is just a local fallback).
• docs/RELEASE.md — release + maintenance documentation (TEM-10).

[kodxana]

Hey, I went through this and I like the general direction, but I’m a bit worried about how the release is coordinated.

Right now each image workflow and release.yml calculates the version independently. If two PRs are merged close together, the second run could start before the first one creates its tag, meaning both runs may calculate the same version and overwrite/race with each other.

The GitHub Release can also be created before all image builds and smoke tests finish, so we could end up with a release where one image family failed or is missing.

Could we have one workflow calculate the version once, pass it to the image builds, wait for them, and only create the release after everything succeeds?

A few smaller things I noticed:

• The path filters miss files like official-templates/base/requirements.txt and official-templates/autoresearch/pre_start.sh.
• Re-running an older release after a newer tag exists may select the newer tag as its base.
• softprops/action-gh-release@v2 should probably be pinned to a commit SHA like the other actions.

Since #142 is based on this branch and #146/#147 touch the same release and versioning files, it would be best to finalize this flow before merging the others.

[kodxana]

Thanks, I checked the latest commits. The release documentation now matches the centralized workflow, and the latest run shows that the new build chain itself is working, all builds and the base, Nvidia, PyTorch, autoresearch, and cluster tests passed.

The ROCm failure looks unrelated to the code: all three images were skipped because no MI300X capacity was available, with 0 actual test failures.

One thing we should decide before merging is whether unavailable ROCm capacity should block a real release on main. With the current on-skip: fail behaviour, this same capacity situation would prevent the release from being created. We may need retries/reserved capacity, or treat a no-capacity skip as a warning.

The main concurrency setup also still keeps only one pending run. This avoids the duplicate-version race, but if three merges land during a long build, the middle release run will be replaced by the newest one. If combining those changes into one release is intentional, that’s fine; otherwise the main release path should use a multi-run queue such as queue: max.

There’s also one small stale comment in nvidia.yml saying that workflow is independently path-filtered and manually dispatchable, which is no longer the case.

[Chmokachka]

@kodxana

The main concurrency setup also still keeps only one pending run. This avoids the duplicate-version race, but if three merges land during a long build, the middle release run will be replaced by the newest one. If combining those changes into one release is intentional, that’s fine; otherwise the main release path should use a multi-run queue such as queue: max.

You're right, but I think it's fine for us. Our version comes from git history (compute-version covers the whole range since the last tag), so even if a middle run gets superseded, the surviving run still includes all those commits - nothing is lost, we'd just get one combined release instead of separate ones. And realistically, this needs 3+ merges within a single long build window, which our small team and merge cadence just don't hit. We can add a merge queue later if per-merge releases ever become a requirement.

All other findings were fixed.

[kodxana]

I noticed one small issue in the successful rerun: the cu1281 cluster image built successfully, but its smoke test didn’t run because downloading cluster-refs-cu1281 returned a 404. The cu1290 and cu1300 smoke tests did run normally.

Because the download step uses continue-on-error, the missing test was silently skipped and the job still passed. It’s probably just a transient GitHub artifact issue, but could we retry or fail visibly here rather than reporting a green test job?

[Madiator2011Work]

LGTM