ci(backend): refuse to move mutable image tags backwards

[localai-bot]

The defect

Backend images publish an immutable per-commit tag (sha-<short>-<suffix>) alongside mutable rolling tags (master-<suffix>, latest-<suffix>, v<version>-<suffix>). Nothing checked whether an incoming build was newer than what a mutable tag already pointed at. Last writer won.

Measured from the live registry:

06:45 UTC 19 Jul  commit 10211948b   (predates the fix below)
07:48 UTC 19 Jul  commit 963c63713   "fix(gpu-libs): bundle cuDNN only where it is used..." (#10946)
10:01 UTC 19 Jul  commit 626ae4d51   (contains the fix)

10:41 UTC 19 Jul  push sha-626ae4d-nvidia-l4t-cuda-13-arm64-longcat-video  digest d4520d47
                  -> master-nvidia-l4t-cuda-13-arm64-longcat-video = CORRECT image
15:40 UTC 19 Jul  push sha-1021194-nvidia-l4t-cuda-13-arm64-longcat-video  digest 38dae6ff
                  -> master-... OVERWRITTEN with the PRE-FIX image

A build of a commit from 06:45 finished roughly nine hours later and moved the mutable tag backwards in time. Every consumer pulling master-* from then on got an image without the cuDNN fix: the pre-fix image bundles a partial 4-of-8 cuDNN library set at 9.24.0 while the venv ships a complete set at 9.20.0.48, giving CUDNN_STATUS_SUBLIBRARY_VERSION_MISMATCH at inference. The fix was built properly and on time, then un-shipped by a straggler, silently, for two days.

Why now: backend CI queues run 4+ hours deep (259.7 min average queue wait vs 18.6 min average execution, peak concurrency 19), and master pushes get a concurrency group keyed by github.sha, so no run supersedes another. Completion order does not track commit order, and the deeper the queue the likelier a straggler clobbers a newer tag.

The defect is still visible in the registry today: quay.io/go-skynet/local-ai-backends:master-cpu-faster-whisper currently carries org.opencontainers.image.revision=10211948b5470d332a93c841a9c8fe2b9a737148.

Why this should merge before or with #11024

#11024 makes GPU backends visible to the upgrade checker for the first time, which will let AutoUpgradeBackends fan upgrades out to worker nodes at scale. With this tag defect unfixed, that machinery would auto-distribute regressed images across the fleet, unattended. This PR should land before or together with #11024.

The fix

An ordering guard: refuse to advance a mutable tag unless the incoming build's commit descends from the commit the tag currently points at.

• scripts/lib/tag-guard.mjs holds the decision logic and the registry/GitHub lookups (injected fetch, so it is testable offline).
• scripts/tag-guard.mjs is the CI entrypoint: reads the docker/metadata-action tag list, prints the tags this build may publish, logs every decision.
• scripts/lib/tag-guard_test.mjs covers the decision table, run by make test-ci-scripts from lint.yml.

Push paths found and covered

backend_build.yml pushes by canonical digest only and applies no tags, so it needed no change. Every mutable tag write happens in one of:

Path	What it writes	Covered
backend_merge.yml, quay imagetools create	multi-arch manifest list and the tagging step for single-arch backends	yes
backend_merge.yml, Docker Hub imagetools create	same, on Docker Hub	yes
backend_build_darwin.yml publish job, crane push loop	darwin tags on both registries	yes

Mechanism chosen, and why

Of the two candidate shapes, this is closest to (2) but reuses stamping that already exists rather than adding it.

• Reading the revision, not mapping digests to sha- tags. docker/metadata-action already emits org.opencontainers.image.revision, and backend_build.yml already passes labels: ${{ steps.meta.outputs.labels }} to build-push-action, so Linux images have carried the source commit all along. Verified against the live registry. The alternative, mapping a digest back to its sha- tag, would mean enumerating a repository with hundreds of thousands of tags on every merge job.
• GitHub compare API, not git merge-base --is-ancestor. The merge and darwin publish jobs use shallow sparse checkouts. merge-base would need full history fetched into each of ~200 merge jobs per push to answer one ancestry question. GET /repos/{repo}/compare/{base}...{head} answers it in one request and needs no git context at all. Both repositories are public, so the registry side works on anonymous pull tokens and the guard needs no credentials of its own.
• Darwin needed the stamp added. Darwin images are crane pushed from a raw OCI tarball and carried no labels at all (verified: master-metal-darwin-arm64-mlx has Labels: null). The publish job now stamps the revision with crane mutate after pushing. Until each darwin tag has been republished once through this step, the guard has nothing to compare and fails open with a warning, which is exactly today's behaviour.

Behaviour

Situation	Outcome
sha-<commit>-* immutable tag	always pushed, unconditionally
mutable tag does not exist yet	pushed, info
incoming commit is ahead / identical	pushed, info
incoming commit is behind	withheld, warning
histories diverged / no ancestor relationship	withheld, warning
image carries no revision label	pushed, warning
published revision no longer reachable in the repo	pushed, warning
registry or compare API error	pushed, warning

Immutable tags always pushing is load-bearing: pinning sha-626ae4d-... is exactly how this incident was worked around, and the block message tells you which sha to pin.

Blocked tags are logged to stderr, raised as ::warning:: annotations and listed in the job summary. The failure mode being fixed here is silence, so a quiet skip would be the same bug in a different costume.

Fail-open on unknown/unresolvable/error is deliberate: a guard that failed closed on a registry or GitHub blip would itself stop shipping merged fixes, which is the bug, only louder. Every fail-open path is annotated.

Release v* tags go through the guard too. Tag builds are inherently ordered and a fresh version tag has never been published, so it takes the "tag does not exist yet" path and always publishes: the guard costs nothing there. It is kept in place because latest-* is emitted by the same builds, and re-running an old release build would otherwise move latest-* backwards.

Validation

Honest summary: the decision logic is tested and the guard has been exercised end to end against the live registry and the live GitHub API, but no real CI build has run this and it cannot be run locally.

Verified:

• make test-ci-scripts passes, 18/18 new tests, exit 0. Includes the incident case (an ancestor commit must not move the tag) and the "immutable tags always survive" invariant.
• Live end-to-end run of scripts/tag-guard.mjs against quay.io/go-skynet/local-ai-backends and localai/localai-backends and the real GitHub compare API, using real commits from this repository:
  • straggler case (078614c, an ancestor of the commit master-cpu-faster-whisper currently carries): withheld the mutable tag on both registries, kept sha-078614c-*, exit 0.
  • newer case (current master tip): 3584e07 descends from 1021194, advancing the tag, plus a never-published v9.9.9-* tag correctly taking the "does not exist yet" path.
• The rewritten shell blocks were run locally with a stubbed docker, confirming the -t argument array, the ci-cache digest glob, REGISTRY_PREFIX filtering, and that first_tag resolves to the immutable tag for the cosign digest lookup.
• A guard crash propagates and fails the step (exit 1) rather than looking like "no tags to push". This is why the workflow writes to a file instead of mapfile < <(...), which would have swallowed the exit status.
• shellcheck clean on all four rewritten run blocks, exit 0. YAML parses.
• crane mutate --label / -t confirmed against crane 0.21.7, the same latest release the darwin job installs.
• backend_pr.yml also calls both reusable workflows; its push-side steps are gated on github.event_name != 'pull_request' and the darwin publish job is gated at job level, so PR builds are unaffected.

Not verified: no real backend CI run has exercised this. Specifically unproven until a build lands: node availability on the runner image (relied on as preinstalled, no setup-node step added), the darwin crane mutate round trip against a real pushed tarball, and the interaction of a withheld tag with the existing Inspect manifest step.

Scope note: the root LocalAI image workflows (image_merge.yml) have the same shape and are not covered here. Worth a follow-up.