Skip to content

feat(hip): migrate every ROCm consumer to 10.0.0 from digest-pinned images (ADR-1225) - #1386

Merged
lusoris merged 3 commits into
masterfrom
feat/rocm-10-migration
Sep 7, 2026
Merged

lusoris merged 3 commits into
masterfrom
feat/rocm-10-migration

Conversation

@lusoris

@lusoris lusoris commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Summary

AMD froze the repo.radeon.com/rocm/apt/ channel at 7.2.4 when ROCm moved to
the "TheRock" build/release system at 7.14 — apt/7.14 and apt/10.0.0 both 404,
the manylinux channel stops at rocm-rel-7.2.4, and the TheRock wheel index carries
only 7.14.0 alphas. The fork was heading into 1.0.0 on a ROCm-legacy toolchain with
no version-string edit available to get off it.

This takes ROCm from the digest-pinned rocm/dev-ubuntu-24.04:10.0.0-full image
instead, across every consumer at once: the dev container, both published GPU
images, and both CI HIP lanes.

Three things ROCm 10 changed that this had to handle — each found by a check
that failed, not by reading release notes:

  1. /opt/rocm/{bin,lib,include,llvm,…} are /etc/alternatives symlinks into
    /opt/rocm/core-10.0/. Copying or extracting /opt/rocm alone lands a directory
    of dangling links; both the rocm-src stage and the extractor repoint them at
    core-10.0/<name> relative to /opt/rocm.
  2. libamdhip64.so's link closure widened to 19 libraries / 234 MB
    (libLLVM, libclang-cpp, libamd_comgr, librocm_kpack,
    librocprofiler-register, the rocm_sysdeps bundle), wired by
    $ORIGIN-relative RPATHs. The node-rocm stage's flat two-library copy no
    longer loads; it now ships the whole closure with its layout intact.
  3. gfx1036 is natively supported, so HSA_OVERRIDE_GFX_VERSION=10.3.0 is dropped
    from dev/docker-compose.yml — keeping it would alias the agent to gfx1030
    while meson compiles gfx1036 code objects.

docker pull is not viable in CI (8.2 GB compressed / 29 GB extracted, on a runner
that already carries CUDA and oneAPI), so scripts/ci/install-rocm-from-image.sh
streams each layer blob from the registry straight into tar, extracting only
/opt/rocm minus the math libraries libvmaf never links. Peak disk is the 5.5 GB
result, not the 29 GB image.

The prune list (19 GB → 5.5 GB) is gated by a hipcc smoke compile in the same
stage. That gate earned its keep: an earlier pass removed librocprofiler-register
with a tidy-looking librocprof* glob, and every HIP binary then failed at load
with cannot open shared object file.

Verification

Measured on the dev host (Linux 7.2.3-1-cachyos, AMD gfx1036), not inferred:

rocminfo                      → Agent 2  Name: gfx1036
amdgpu-arch                   → gfx1036            (native, no HSA override)
hipcc --offload-arch=gfx1036  → compiles; kernel dispatches (h[63] = 63)
libvmaf build                 → 1537/1537 targets
meson test (HIP suite)        → Ok: 19, Expected Fail: 4, Fail: 0
end-to-end                    → hip vmaf = 45.315104 / cpu vmaf = 45.315104

Container stage, built and run:

rocm-src prune                 → 19 GB → 5.5 GB, hipcc still emits a runnable binary
COPY into an ubuntu:26.04 stage → hipcc gfx1036 OK, no dangling top-level symlinks
node-rocm copy set             → ldd libamdhip64.so: 0 unresolved, 397 MB
extractor (registry → tarball) → 5.5 GB, hipcc gfx1036 compiles and the binary runs
extractor (multi-arch index)   → resolves linux/amd64, clean error on a non-ROCm image

Reproducer / smoke-test command

# Extract ROCm 10 from the pinned image on any Linux host with curl + jq + tar
scripts/ci/install-rocm-from-image.sh \
  --image rocm/dev-ubuntu-24.04@sha256:a90cf047f615abe70fbef83c64def0a2d549ef37a39c8ea545430aba4981b374 \
  --dest /tmp/rocm10

printf '#include <hip/hip_runtime.h>\n#include <cstdio>\n__global__ void k(int*o){o[threadIdx.x]=threadIdx.x;}\nint main(){printf("ok\\n");return 0;}\n' > /tmp/smoke.hip
ROCM_PATH=/tmp/rocm10 LD_LIBRARY_PATH=/tmp/rocm10/lib \
  /tmp/rocm10/bin/hipcc --offload-arch=gfx1036 /tmp/smoke.hip -o /tmp/smoke && /tmp/smoke

# Dev container, end to end
docker compose --project-directory "$(git rev-parse --show-toplevel)" \
  -f dev/docker-compose.yml build dev-mcp
docker exec vmaf-dev-mcp vmaf \
  --reference /workspace/python/test/resource/yuv/src01_hrc00_576x324.yuv \
  --distorted /workspace/python/test/resource/yuv/src01_hrc01_576x324.yuv \
  --width 576 --height 324 --pixel_format 420 --bitdepth 8 --backend hip

Deep-dive deliverables (ADR-0108)

  • Research digest — docs/research/1225-rocm-10-distribution-and-layout.md: the channel survey, the alternatives layout, the link closure, the prune measurements, and why CI streams instead of pulling.
  • Decision matrix — ADR-1225 ## Alternatives considered (six options, including the deferred GHCR mirror).
  • AGENTS.md invariant note — dev/AGENTS.md (never restore the apt path; never prune librocprofiler-register; HSA_OVERRIDE_GFX_VERSION stays out) and the root AGENTS.md container-pin section.
  • Reproducer / smoke-test command — above.
  • Changelog fragment — changelog.d/changed/1225-rocm-10-therock-migration.md, CHANGELOG.md regenerated.
  • Rebase note — docs/rebase-notes.md § feat/rocm-10-migration (ADR-1225), five invariants.

Docs (rule 10)

State (rule 13)

no state delta: dependency migration — no bug opened, closed, or ruled not-affecting.

ffmpeg-patches (rule 14)

no patch impact: no C-API surface, CLI flag, meson_options.txt entry or public
header changes; the meson edit is comment-only.

🤖 Generated with Claude Code

@lusoris
lusoris force-pushed the feat/rocm-10-migration branch 2 times, most recently from 20bdf88 to a334256 Compare September 7, 2026 12:16
lusoris and others added 2 commits September 7, 2026 17:41
…mages

AMD froze the repo.radeon.com apt channel at 7.2.4 when ROCm moved to the
"TheRock" build/release system at 7.14: apt/7.14 and apt/10.0.0 both 404,
the manylinux channel stops at rocm-rel-7.2.4, and the TheRock wheel index
carries only 7.14.0 alphas. The fork was therefore stuck on a ROCm-legacy
toolchain heading into 1.0.0, with no version-string edit available to get
off it.

Take ROCm from the digest-pinned rocm/dev-ubuntu-24.04:10.0.0-full image
instead. dev/Containerfile, docker/Dockerfile.production-gpu and
docker/Dockerfile.node copy a pruned /opt/rocm out of it; the two CI HIP
lanes use the new scripts/ci/install-rocm-from-image.sh, which streams the
image's /opt/rocm out of the registry layer by layer. A docker pull is not
viable on a runner that already carries CUDA and oneAPI -- 8.2 GB
compressed, 29 GB extracted -- so nothing is ever stored whole: peak disk
is the 5.5 GB result.

Three things ROCm 10 changed that this had to handle:

  - /opt/rocm/{bin,lib,include,llvm,...} are now /etc/alternatives symlinks
    into /opt/rocm/core-10.0/. Copying or extracting /opt/rocm alone lands
    dangling links, so both the rocm-src stage and the extractor repoint
    them at core-10.0/<name> relative to /opt/rocm.
  - libamdhip64.so's link closure widened to 19 libraries (234 MB) wired by
    $ORIGIN-relative RPATHs. The node-rocm runtime stage's flat copy of
    libamdhip64.so* + libhsa-runtime64.so* no longer loads; it now ships
    the whole closure with its layout intact, verified at 0 unresolved
    libraries in a scratch stage carrying only the copied files.
  - gfx1036 is natively supported, so HSA_OVERRIDE_GFX_VERSION=10.3.0 is
    dropped from dev/docker-compose.yml. Keeping it would alias the agent
    to gfx1030 while meson compiles gfx1036 code objects.

The prune list (19 GB -> 5.5 GB) is gated by a hipcc smoke compile in the
same stage: an earlier pass removed librocprofiler-register with a tidy
librocprof* glob and every HIP binary then failed at load.

Renovate follows the image digest now rather than scraping the frozen apt
directory listing.

Verified on AMD gfx1036 under Linux 7.2.3: HIP suite 19 Ok / 0 Fail, and an
end-to-end HIP score of 45.315104, bit-identical to the CPU reference.

Refs: ADR-1225

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The badge scraped `ARG ROCM_VER=` out of dev/Containerfile, which this
branch removes along with the apt install. Read the version out of the
digest-pinned rocm/dev-ubuntu-24.04:<version>-full reference instead, so it
keeps reporting a live value rather than going blank.

Verified by running the badge's own regex against the new Containerfile:
matches `rocm/dev-ubuntu-24.04:10.0.0-full`, renders `10.0.0`.

Refs: ADR-1225

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@lusoris
lusoris force-pushed the feat/rocm-10-migration branch from a334256 to f482628 Compare September 7, 2026 15:41
@lusoris
lusoris marked this pull request as ready for review September 7, 2026 15:43
@lusoris
lusoris enabled auto-merge (squash) September 7, 2026 15:43
The docker-publish source-binding test hardcoded the full job names, including
the vendor SDK generation: `build-rocm7`, `build-oneapi2025`. Renaming a job as
part of an SDK bump therefore failed the Release Script Contract with
`AssertionError: missing job build-rocm7` — a failure about the rename, not
about anything the test actually checks. This branch's `build-rocm7` ->
`build-rocm10` tripped it, and the pending oneAPI 2026 bump would have tripped
it again the same way.

Match the stable prefix instead and resolve the real job name from the
workflow. The assertion that matters is unchanged: exactly one such build job
exists and is bound to the published tag. An ambiguous prefix is now an
explicit error rather than a silent first-match, and a genuinely missing job
still fails (`missing job matching build-rocm*`).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@lusoris
lusoris merged commit 5143e5d into master Sep 7, 2026
77 checks passed
@lusoris
lusoris deleted the feat/rocm-10-migration branch September 7, 2026 16:17
@lusoris lusoris added the type:feature New feature or request label Sep 7, 2026
lusoris added a commit that referenced this pull request Sep 7, 2026
…ADR-1231)

Container base images were pinned by hand at each point of use and drifted.
`Dockerfile.controller` and `Dockerfile.operator` still built on Debian 12 and
shipped on distroless-debian12 after the rest of the tree moved to Debian 13,
and both quoted a golang digest in their header comments that did not match
their own FROM line. Two CUDA pins sat on Ubuntu 24.04 beside a sibling stage
on 26.04.

Four further pins were invisible to any FROM scan because they lived in
`COPY --from=<image>` -- the Go toolchain in dev/Containerfile and the CUDA,
ROCm and oneAPI runtime libraries in Dockerfile.node. Those were the most stale
images in the repository, which is the argument both for gating `COPY --from`
and for converting them to named stages.

build-config.env is now the single source of truth. No Dockerfile names a base
directly: each takes it as an ARG whose default mirrors the config, so a plain
`docker build` still works with no wrapper and CI can override any base with
--build-arg. `scripts/ci/check-base-image-single-source.sh` fails on drift, on
a pin that is not digest-pinned, and on a version knob that disagrees with its
pin; `make base-images-sync` rewrites the mirrors. Same authority-plus-drift-
check shape as check-default-model-single-source.sh.

Removes four Debian 12 pins and two Ubuntu 24.04 CUDA pins. `vmafx-controller`
gains an explicit `USER 65532:65532`: its old cc-debian12 pin lacked the
`:nonroot` suffix, so it ran as root. Both its listen ports are above 1024.

ROCm and oneAPI keep Ubuntu 24.04 under explicit, self-closing exemptions.
Each is a major SDK migration needing matching source changes, not a pin swap:
ROCm 7.2.4 -> 10.0.0 is PR #1386, and oneAPI 2025 -> 2026.1 is blocked on a
libsycl soname bump (.so.8 -> .so.9), a glibc gap between Intel's Ubuntu images
and Debian 13, and the NEO GPU driver that Intel's runtime image carries and
Debian does not package. All measured; see the research digest.

Also fixes a latent bug in dev/Containerfile: `ENV PYTHONPATH=/build/vmaf/python:${PYTHONPATH}`
expanded to a trailing colon because PYTHONPATH was never set, putting the
current directory on sys.path.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
lusoris added a commit that referenced this pull request Sep 8, 2026
…ADR-1231)

Container base images were pinned by hand at each point of use and drifted.
`Dockerfile.controller` and `Dockerfile.operator` still built on Debian 12 and
shipped on distroless-debian12 after the rest of the tree moved to Debian 13,
and both quoted a golang digest in their header comments that did not match
their own FROM line. Two CUDA pins sat on Ubuntu 24.04 beside a sibling stage
on 26.04.

Four further pins were invisible to any FROM scan because they lived in
`COPY --from=<image>` -- the Go toolchain in dev/Containerfile and the CUDA,
ROCm and oneAPI runtime libraries in Dockerfile.node. Those were the most stale
images in the repository, which is the argument both for gating `COPY --from`
and for converting them to named stages.

build-config.env is now the single source of truth. No Dockerfile names a base
directly: each takes it as an ARG whose default mirrors the config, so a plain
`docker build` still works with no wrapper and CI can override any base with
--build-arg. `scripts/ci/check-base-image-single-source.sh` fails on drift, on
a pin that is not digest-pinned, and on a version knob that disagrees with its
pin; `make base-images-sync` rewrites the mirrors. Same authority-plus-drift-
check shape as check-default-model-single-source.sh.

Removes four Debian 12 pins and two Ubuntu 24.04 CUDA pins. `vmafx-controller`
gains an explicit `USER 65532:65532`: its old cc-debian12 pin lacked the
`:nonroot` suffix, so it ran as root. Both its listen ports are above 1024.

ROCm and oneAPI keep Ubuntu 24.04 under explicit, self-closing exemptions.
Each is a major SDK migration needing matching source changes, not a pin swap:
ROCm 7.2.4 -> 10.0.0 is PR #1386, and oneAPI 2025 -> 2026.1 is blocked on a
libsycl soname bump (.so.8 -> .so.9), a glibc gap between Intel's Ubuntu images
and Debian 13, and the NEO GPU driver that Intel's runtime image carries and
Debian does not package. All measured; see the research digest.

Also fixes a latent bug in dev/Containerfile: `ENV PYTHONPATH=/build/vmaf/python:${PYTHONPATH}`
expanded to a trailing colon because PYTHONPATH was never set, putting the
current directory on sys.path.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
lusoris added a commit that referenced this pull request Sep 8, 2026
…ADR-1231)

Container base images were pinned by hand at each point of use and drifted.
`Dockerfile.controller` and `Dockerfile.operator` still built on Debian 12 and
shipped on distroless-debian12 after the rest of the tree moved to Debian 13,
and both quoted a golang digest in their header comments that did not match
their own FROM line. Two CUDA pins sat on Ubuntu 24.04 beside a sibling stage
on 26.04.

Four further pins were invisible to any FROM scan because they lived in
`COPY --from=<image>` -- the Go toolchain in dev/Containerfile and the CUDA,
ROCm and oneAPI runtime libraries in Dockerfile.node. Those were the most stale
images in the repository, which is the argument both for gating `COPY --from`
and for converting them to named stages.

build-config.env is now the single source of truth. No Dockerfile names a base
directly: each takes it as an ARG whose default mirrors the config, so a plain
`docker build` still works with no wrapper and CI can override any base with
--build-arg. `scripts/ci/check-base-image-single-source.sh` fails on drift, on
a pin that is not digest-pinned, and on a version knob that disagrees with its
pin; `make base-images-sync` rewrites the mirrors. Same authority-plus-drift-
check shape as check-default-model-single-source.sh.

Removes four Debian 12 pins and two Ubuntu 24.04 CUDA pins. `vmafx-controller`
gains an explicit `USER 65532:65532`: its old cc-debian12 pin lacked the
`:nonroot` suffix, so it ran as root. Both its listen ports are above 1024.

ROCm and oneAPI keep Ubuntu 24.04 under explicit, self-closing exemptions.
Each is a major SDK migration needing matching source changes, not a pin swap:
ROCm 7.2.4 -> 10.0.0 is PR #1386, and oneAPI 2025 -> 2026.1 is blocked on a
libsycl soname bump (.so.8 -> .so.9), a glibc gap between Intel's Ubuntu images
and Debian 13, and the NEO GPU driver that Intel's runtime image carries and
Debian does not package. All measured; see the research digest.

Also fixes a latent bug in dev/Containerfile: `ENV PYTHONPATH=/build/vmaf/python:${PYTHONPATH}`
expanded to a trailing colon because PYTHONPATH was never set, putting the
current directory on sys.path.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
lusoris added a commit that referenced this pull request Sep 8, 2026
…ADR-1231)

Container base images were pinned by hand at each point of use and drifted.
`Dockerfile.controller` and `Dockerfile.operator` still built on Debian 12 and
shipped on distroless-debian12 after the rest of the tree moved to Debian 13,
and both quoted a golang digest in their header comments that did not match
their own FROM line. Two CUDA pins sat on Ubuntu 24.04 beside a sibling stage
on 26.04.

Four further pins were invisible to any FROM scan because they lived in
`COPY --from=<image>` -- the Go toolchain in dev/Containerfile and the CUDA,
ROCm and oneAPI runtime libraries in Dockerfile.node. Those were the most stale
images in the repository, which is the argument both for gating `COPY --from`
and for converting them to named stages.

build-config.env is now the single source of truth. No Dockerfile names a base
directly: each takes it as an ARG whose default mirrors the config, so a plain
`docker build` still works with no wrapper and CI can override any base with
--build-arg. `scripts/ci/check-base-image-single-source.sh` fails on drift, on
a pin that is not digest-pinned, and on a version knob that disagrees with its
pin; `make base-images-sync` rewrites the mirrors. Same authority-plus-drift-
check shape as check-default-model-single-source.sh.

Removes four Debian 12 pins and two Ubuntu 24.04 CUDA pins. `vmafx-controller`
gains an explicit `USER 65532:65532`: its old cc-debian12 pin lacked the
`:nonroot` suffix, so it ran as root. Both its listen ports are above 1024.

ROCm and oneAPI keep Ubuntu 24.04 under explicit, self-closing exemptions.
Each is a major SDK migration needing matching source changes, not a pin swap:
ROCm 7.2.4 -> 10.0.0 is PR #1386, and oneAPI 2025 -> 2026.1 is blocked on a
libsycl soname bump (.so.8 -> .so.9), a glibc gap between Intel's Ubuntu images
and Debian 13, and the NEO GPU driver that Intel's runtime image carries and
Debian does not package. All measured; see the research digest.

Also fixes a latent bug in dev/Containerfile: `ENV PYTHONPATH=/build/vmaf/python:${PYTHONPATH}`
expanded to a trailing colon because PYTHONPATH was never set, putting the
current directory on sys.path.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
lusoris added a commit that referenced this pull request Sep 8, 2026
…ADR-1231)

Container base images were pinned by hand at each point of use and drifted.
`Dockerfile.controller` and `Dockerfile.operator` still built on Debian 12 and
shipped on distroless-debian12 after the rest of the tree moved to Debian 13,
and both quoted a golang digest in their header comments that did not match
their own FROM line. Two CUDA pins sat on Ubuntu 24.04 beside a sibling stage
on 26.04.

Four further pins were invisible to any FROM scan because they lived in
`COPY --from=<image>` -- the Go toolchain in dev/Containerfile and the CUDA,
ROCm and oneAPI runtime libraries in Dockerfile.node. Those were the most stale
images in the repository, which is the argument both for gating `COPY --from`
and for converting them to named stages.

build-config.env is now the single source of truth. No Dockerfile names a base
directly: each takes it as an ARG whose default mirrors the config, so a plain
`docker build` still works with no wrapper and CI can override any base with
--build-arg. `scripts/ci/check-base-image-single-source.sh` fails on drift, on
a pin that is not digest-pinned, and on a version knob that disagrees with its
pin; `make base-images-sync` rewrites the mirrors. Same authority-plus-drift-
check shape as check-default-model-single-source.sh.

Removes four Debian 12 pins and two Ubuntu 24.04 CUDA pins. `vmafx-controller`
gains an explicit `USER 65532:65532`: its old cc-debian12 pin lacked the
`:nonroot` suffix, so it ran as root. Both its listen ports are above 1024.

ROCm and oneAPI keep Ubuntu 24.04 under explicit, self-closing exemptions.
Each is a major SDK migration needing matching source changes, not a pin swap:
ROCm 7.2.4 -> 10.0.0 is PR #1386, and oneAPI 2025 -> 2026.1 is blocked on a
libsycl soname bump (.so.8 -> .so.9), a glibc gap between Intel's Ubuntu images
and Debian 13, and the NEO GPU driver that Intel's runtime image carries and
Debian does not package. All measured; see the research digest.

Also fixes a latent bug in dev/Containerfile: `ENV PYTHONPATH=/build/vmaf/python:${PYTHONPATH}`
expanded to a trailing colon because PYTHONPATH was never set, putting the
current directory on sys.path.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
lusoris added a commit that referenced this pull request Sep 8, 2026
…ADR-1231)

Container base images were pinned by hand at each point of use and drifted.
`Dockerfile.controller` and `Dockerfile.operator` still built on Debian 12 and
shipped on distroless-debian12 after the rest of the tree moved to Debian 13,
and both quoted a golang digest in their header comments that did not match
their own FROM line. Two CUDA pins sat on Ubuntu 24.04 beside a sibling stage
on 26.04.

Four further pins were invisible to any FROM scan because they lived in
`COPY --from=<image>` -- the Go toolchain in dev/Containerfile and the CUDA,
ROCm and oneAPI runtime libraries in Dockerfile.node. Those were the most stale
images in the repository, which is the argument both for gating `COPY --from`
and for converting them to named stages.

build-config.env is now the single source of truth. No Dockerfile names a base
directly: each takes it as an ARG whose default mirrors the config, so a plain
`docker build` still works with no wrapper and CI can override any base with
--build-arg. `scripts/ci/check-base-image-single-source.sh` fails on drift, on
a pin that is not digest-pinned, and on a version knob that disagrees with its
pin; `make base-images-sync` rewrites the mirrors. Same authority-plus-drift-
check shape as check-default-model-single-source.sh.

Removes four Debian 12 pins and two Ubuntu 24.04 CUDA pins. `vmafx-controller`
gains an explicit `USER 65532:65532`: its old cc-debian12 pin lacked the
`:nonroot` suffix, so it ran as root. Both its listen ports are above 1024.

ROCm and oneAPI keep Ubuntu 24.04 under explicit, self-closing exemptions.
Each is a major SDK migration needing matching source changes, not a pin swap:
ROCm 7.2.4 -> 10.0.0 is PR #1386, and oneAPI 2025 -> 2026.1 is blocked on a
libsycl soname bump (.so.8 -> .so.9), a glibc gap between Intel's Ubuntu images
and Debian 13, and the NEO GPU driver that Intel's runtime image carries and
Debian does not package. All measured; see the research digest.

Also fixes a latent bug in dev/Containerfile: `ENV PYTHONPATH=/build/vmaf/python:${PYTHONPATH}`
expanded to a trailing colon because PYTHONPATH was never set, putting the
current directory on sys.path.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
lusoris added a commit that referenced this pull request Sep 8, 2026
…ADR-1231)

Container base images were pinned by hand at each point of use and drifted.
`Dockerfile.controller` and `Dockerfile.operator` still built on Debian 12 and
shipped on distroless-debian12 after the rest of the tree moved to Debian 13,
and both quoted a golang digest in their header comments that did not match
their own FROM line. Two CUDA pins sat on Ubuntu 24.04 beside a sibling stage
on 26.04.

Four further pins were invisible to any FROM scan because they lived in
`COPY --from=<image>` -- the Go toolchain in dev/Containerfile and the CUDA,
ROCm and oneAPI runtime libraries in Dockerfile.node. Those were the most stale
images in the repository, which is the argument both for gating `COPY --from`
and for converting them to named stages.

build-config.env is now the single source of truth. No Dockerfile names a base
directly: each takes it as an ARG whose default mirrors the config, so a plain
`docker build` still works with no wrapper and CI can override any base with
--build-arg. `scripts/ci/check-base-image-single-source.sh` fails on drift, on
a pin that is not digest-pinned, and on a version knob that disagrees with its
pin; `make base-images-sync` rewrites the mirrors. Same authority-plus-drift-
check shape as check-default-model-single-source.sh.

Removes four Debian 12 pins and two Ubuntu 24.04 CUDA pins. `vmafx-controller`
gains an explicit `USER 65532:65532`: its old cc-debian12 pin lacked the
`:nonroot` suffix, so it ran as root. Both its listen ports are above 1024.

ROCm and oneAPI keep Ubuntu 24.04 under explicit, self-closing exemptions.
Each is a major SDK migration needing matching source changes, not a pin swap:
ROCm 7.2.4 -> 10.0.0 is PR #1386, and oneAPI 2025 -> 2026.1 is blocked on a
libsycl soname bump (.so.8 -> .so.9), a glibc gap between Intel's Ubuntu images
and Debian 13, and the NEO GPU driver that Intel's runtime image carries and
Debian does not package. All measured; see the research digest.

Also fixes a latent bug in dev/Containerfile: `ENV PYTHONPATH=/build/vmaf/python:${PYTHONPATH}`
expanded to a trailing colon because PYTHONPATH was never set, putting the
current directory on sys.path.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
lusoris added a commit that referenced this pull request Sep 8, 2026
…ADR-1231)

Container base images were pinned by hand at each point of use and drifted.
`Dockerfile.controller` and `Dockerfile.operator` still built on Debian 12 and
shipped on distroless-debian12 after the rest of the tree moved to Debian 13,
and both quoted a golang digest in their header comments that did not match
their own FROM line. Two CUDA pins sat on Ubuntu 24.04 beside a sibling stage
on 26.04.

Four further pins were invisible to any FROM scan because they lived in
`COPY --from=<image>` -- the Go toolchain in dev/Containerfile and the CUDA,
ROCm and oneAPI runtime libraries in Dockerfile.node. Those were the most stale
images in the repository, which is the argument both for gating `COPY --from`
and for converting them to named stages.

build-config.env is now the single source of truth. No Dockerfile names a base
directly: each takes it as an ARG whose default mirrors the config, so a plain
`docker build` still works with no wrapper and CI can override any base with
--build-arg. `scripts/ci/check-base-image-single-source.sh` fails on drift, on
a pin that is not digest-pinned, and on a version knob that disagrees with its
pin; `make base-images-sync` rewrites the mirrors. Same authority-plus-drift-
check shape as check-default-model-single-source.sh.

Removes four Debian 12 pins and two Ubuntu 24.04 CUDA pins. `vmafx-controller`
gains an explicit `USER 65532:65532`: its old cc-debian12 pin lacked the
`:nonroot` suffix, so it ran as root. Both its listen ports are above 1024.

ROCm and oneAPI keep Ubuntu 24.04 under explicit, self-closing exemptions.
Each is a major SDK migration needing matching source changes, not a pin swap:
ROCm 7.2.4 -> 10.0.0 is PR #1386, and oneAPI 2025 -> 2026.1 is blocked on a
libsycl soname bump (.so.8 -> .so.9), a glibc gap between Intel's Ubuntu images
and Debian 13, and the NEO GPU driver that Intel's runtime image carries and
Debian does not package. All measured; see the research digest.

Also fixes a latent bug in dev/Containerfile: `ENV PYTHONPATH=/build/vmaf/python:${PYTHONPATH}`
expanded to a trailing colon because PYTHONPATH was never set, putting the
current directory on sys.path.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
lusoris added a commit that referenced this pull request Sep 15, 2026
…ADR-1231) (#1396)

* build(docker): define every container base image in one config file (ADR-1231)

Container base images were pinned by hand at each point of use and drifted.
`Dockerfile.controller` and `Dockerfile.operator` still built on Debian 12 and
shipped on distroless-debian12 after the rest of the tree moved to Debian 13,
and both quoted a golang digest in their header comments that did not match
their own FROM line. Two CUDA pins sat on Ubuntu 24.04 beside a sibling stage
on 26.04.

Four further pins were invisible to any FROM scan because they lived in
`COPY --from=<image>` -- the Go toolchain in dev/Containerfile and the CUDA,
ROCm and oneAPI runtime libraries in Dockerfile.node. Those were the most stale
images in the repository, which is the argument both for gating `COPY --from`
and for converting them to named stages.

build-config.env is now the single source of truth. No Dockerfile names a base
directly: each takes it as an ARG whose default mirrors the config, so a plain
`docker build` still works with no wrapper and CI can override any base with
--build-arg. `scripts/ci/check-base-image-single-source.sh` fails on drift, on
a pin that is not digest-pinned, and on a version knob that disagrees with its
pin; `make base-images-sync` rewrites the mirrors. Same authority-plus-drift-
check shape as check-default-model-single-source.sh.

Removes four Debian 12 pins and two Ubuntu 24.04 CUDA pins. `vmafx-controller`
gains an explicit `USER 65532:65532`: its old cc-debian12 pin lacked the
`:nonroot` suffix, so it ran as root. Both its listen ports are above 1024.

ROCm and oneAPI keep Ubuntu 24.04 under explicit, self-closing exemptions.
Each is a major SDK migration needing matching source changes, not a pin swap:
ROCm 7.2.4 -> 10.0.0 is PR #1386, and oneAPI 2025 -> 2026.1 is blocked on a
libsycl soname bump (.so.8 -> .so.9), a glibc gap between Intel's Ubuntu images
and Debian 13, and the NEO GPU driver that Intel's runtime image carries and
Debian does not package. All measured; see the research digest.

Also fixes a latent bug in dev/Containerfile: `ENV PYTHONPATH=/build/vmaf/python:${PYTHONPATH}`
expanded to a trailing colon because PYTHONPATH was never set, putting the
current directory on sys.path.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* build(config): extend the single source to toolchain versions and Renovate

The first commit unified container base images. That was half the problem: the
same versions live in CI workflows, and they had drifted there independently.

ROCm was 7.2.4 in build.yml and 7.2.3 in libvmaf-build-matrix.yml, so CI
validated a ROCm the published images never shipped. The Level Zero loader
existed at FOUR versions simultaneously -- v1.18.5, v1.28.0 twice, v1.29.0 and
1.32.0 -- and a skew there does not fail a build. It surfaces at runtime as
"No device of requested type available", which reads like a driver or
permissions problem and is not.

build-config.env now carries the toolchain versions too (ROCM_APT_VERSION,
LEVEL_ZERO_VERSION, CUDA_APT_PACKAGE, PYTHON_CI_VERSION). Workflow `run:` steps
source it directly; scripts/ci/load-build-config.sh exports every knob into
$GITHUB_ENV for steps that need a value earlier than that. The Windows SYCL leg
runs under cmd and cannot source it, so it mirrors the value by hand and the
gate keeps that mirror honest.

scripts/ci/check-workflow-versions.py enforces it. It is a separate script
because the Level Zero clone puts `--branch vX.Y.Z` and the repository URL on
different lines, so a line-oriented grep silently sees only the first line --
the first version of this check passed against a deliberately planted literal.

Renovate is reconfigured to match. Its `dockerfile` manager understands
`ARG X=image` + `FROM $X`, so left alone it would bump the Dockerfile mirrors
and leave build-config.env behind -- failing this change's own gate on
Renovate's PRs. One custom manager now matches both the config line and the ARG
form across all nine wired files (41 pins), so a bump lands every copy in one
PR, and a packageRule disables the built-in manager on exactly those files.
docker/dev/*.Dockerfile deliberately keeps the built-in manager.

The gate also classifies config keys by the SHAPE of their value rather than by
a list of names, so adding a version knob cannot accidentally subject it to the
digest-pinning rule.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* build(rocm): move the ROCm images to the Ubuntu 26.04 variant of 10.0.0

ADR-1225 landed ROCm 10.0.0 on the ubuntu-24.04 variant, which left the tree
with the last release images on a distro everything else had moved off.

AMD publishes rocm/dev-ubuntu-26.04:10.0.0-full too, but this is not a pin swap
to make casually: Dockerfile.node and Dockerfile.production-gpu COPY these
libraries out of the vendor image into a Debian 13 runtime, so a libc mismatch
does not fail the build -- it fails the load, on the user's machine, with a
"cannot open shared object file" that points at the wrong thing.

Measured before moving:

  /opt/rocm/core-10.0/lib layout   identical in both variants
  max GLIBC symbol in the closure  2.28 in both (AMD builds to an old floor)
  Debian 13 runtime provides       2.41

So the 26.04 variant is a drop-in. This is the OPPOSITE of the oneAPI case,
where Intel's Ubuntu 26.04 image requires glibc 2.43 and genuinely cannot be
copied onto Debian 13 -- the question has to be asked per vendor rather than
answered once for all of them.

With ROCm current, the gate's Ubuntu-24.04 exemption list is down to the two
oneAPI entries, which disappear with the 2026.1 migration.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* build(ci): single-source the oneAPI version and move Linux to 2026.1

build-config.env was added to stop version drift, but the oneAPI knob was
deliberately left out with a comment saying the 2026.1 migration was "the
immediate follow-up to this file". That follow-up never happened, so three
workflows kept spelling `intel-oneapi-compiler-dpcpp-cpp-2025.3` inline and the
CI toolchain sat two majors behind the compiler installed on the workstation.

All three already sourced build-config.env for LEVEL_ZERO_VERSION, so the
plumbing existed; only the version was missing from it. They now read
${ONEAPI_APT_PACKAGE}, and ONEAPI_VERSION is 2026.1.

The config also gains ONEAPI_RUNTIME_APT_PACKAGES, which names
intel-oneapi-umf explicitly: intel-oneapi-runtime-dpcpp-cpp does not depend on
it, so libumf.so.1 goes missing and the adapter fails to load -- surfacing as
"No device of requested type available" rather than as a link error. That was
measured while preparing the migration, and is the kind of fact the single
source exists to keep.

The Windows leg stays at 2025.3.0.372 on purpose. Its offline-installer URL
carries an opaque per-build GUID that cannot be derived from a version number,
so bumping it needs a looked-up URL rather than a string substitution; it is
recorded as ONEAPI_WINDOWS_VERSION so the lag is visible in the same file
instead of buried in a workflow, and tracked as
T-ONEAPI-WINDOWS-CI-LAG-2026-09-07.

Verified: load-build-config.sh resolves ONEAPI_APT_PACKAGE to
intel-oneapi-compiler-dpcpp-cpp-2026.1, no literal 2025.3 oneAPI package name
remains in .github/workflows/, and both check-workflow-versions.py and
check-base-image-single-source.sh pass.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(rocm): restore valid Ubuntu 24.04 ROCm 10 image pin from ADR-1225

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(ci): strip tag component from repository name in install-rocm-from-image

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(ci): format check-workflow-versions.py with black

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: repair RC1 hooks, patch maintenance and repository hygiene (#1420)

* fix(ffmpeg): replay percentile guard after the existing mappings

* docs(adr): define stable-release FFmpeg patch maintenance

* fix(ffmpeg): automate stable-release patch refresh and required replay

* fix(go): confine predictor cards and require Go validation

* docs(adr): preserve work during agent-state cleanup

* fix(dev): preserve work during agent-state cleanup

* fix(ci): recognize root build config in dependency PRs

* fix(hooks): preserve dispatch across worktree removal

* fix(ci): reject unpinned container image bypasses

* fix(docs): refresh ADR metadata and enforce source freshness

* fix(ci): align shared config consumers and combined documentation

* fix(deps): restore Renovate custom manager file matching

* fix(docs): require push toolchain and repair guide links

* test(ffmpeg): type patch receipts and safety fixtures for push checks

* test(ci): type impact and Go workflow contracts

* docs: repair topic links and identify unavailable evidence

* fix(build): consume the shared Level Zero version

* docs: refresh FFmpeg guide and correct PSNR evidence

* fix(dev): make container shell failures explicit

* fix(hooks): retain documentation checks on large diffs

---------

Co-authored-by: Lusoris <lusoris@pm.me>

* fix(docker): un-ignore build-config.env, which build*/ was silently excluding

The Dev Container build failed on this PR with

    failed to compute cache key: "/build-config.env": not found

even though `build-config.env` sits at the repository root, is tracked, and no
`.dockerignore` pattern names it. Docker cleans a trailing slash off an ignore
pattern, so `build*/` is matched as `build*`, which also matches the new
root-level *file* this PR introduces, and the file never enters the build
context. `COPY build-config.env` then cannot find it.

Reproduced from first principles rather than inferred: a two-line
`.dockerignore` containing only `build*/` and `build-*/`, plus a Dockerfile whose
sole instruction is `COPY build-config.env`, fails with that exact message on
Docker 29.8.0. Adding `!build-config.env` makes it succeed, and a follow-up build
confirms `build-cpu/` is still excluded, so the directory patterns keep working.

This is the last thing standing between the single-source config and a green
container build — which matters beyond this PR, because master's `Docker Image
Build` has been red since 2026-09-08 on the FFmpeg patch series that this branch
also repairs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(docker): drop NVIDIA's malformed CUDA apt list before installing

`Docker Image Build` fails at the first `apt-get install` with

    E: Encountered a section with no Package: header
    E: Problem with MergeList /var/lib/apt/lists/developer.download.nvidia.com_
       compute_cuda_repos_ubuntu2604_x86%5f64_Packages.lz4

NVIDIA's ubuntu2604 CUDA index is currently malformed. Verified as upstream and
not ours by running the pinned digest directly — `docker run
nvidia/cuda:13.3.1-devel-ubuntu26.04@sha256:8cf42b8d… apt-get update` reproduces
it with no repository content involved — and the digest is byte-identical on
master and on this branch, so the base image did not change here.

None of the packages installed in that step come from the CUDA repo, and the
toolkit is baked into the image rather than installed from it: after removing the
list, `nvcc --version` still reports `cuda_13.3.r13.3`, and the install of
build-essential, ccache, ninja-build, nasm, python3 and pkg-config succeeds. The
whole first stage now builds locally to a clean export.

This is the same remedy `.github/workflows/go-ci.yml` already applies to the
Microsoft and Azure sources on hosted runners, for exactly this class of
third-party-index breakage.

Second of the two container failures on this branch. The first was the FFmpeg
patch series: patch 0018 stopped applying at `vf_libvmaf.c:940` once percentile
pooling landed twice, which is what has kept master's `Docker Image Build` and
`FFmpeg SYCL` red since 2026-09-08, and which the 0018 refresh already carried
here repairs — `ffmpeg_patch_stack.py --check` replays all 18 patches onto n9.0.1.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Lusoris <lusoris@pm.me>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type:feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant