Repository navigation
build(docker): define every container base image in one config file (ADR-1231) - #1396
Merged
Merged
Conversation
lusoris
force-pushed
the
build/base-image-single-source
branch
2 times, most recently
from
September 8, 2026 12:29
7de8114 to
280d012
Compare
lusoris
marked this pull request as ready for review
September 8, 2026 12:30
lusoris
enabled auto-merge (squash)
September 8, 2026 12:31
lusoris
force-pushed
the
build/base-image-single-source
branch
5 times, most recently
from
September 8, 2026 15:34
51fb602 to
ea11584
Compare
9 of 10 tasks
…ADR-1231) Container base images were pinned by hand at each point of use and drifted. `Dockerfile.controller` and `Dockerfile.operator` still built on Debian 12 and shipped on distroless-debian12 after the rest of the tree moved to Debian 13, and both quoted a golang digest in their header comments that did not match their own FROM line. Two CUDA pins sat on Ubuntu 24.04 beside a sibling stage on 26.04. Four further pins were invisible to any FROM scan because they lived in `COPY --from=<image>` -- the Go toolchain in dev/Containerfile and the CUDA, ROCm and oneAPI runtime libraries in Dockerfile.node. Those were the most stale images in the repository, which is the argument both for gating `COPY --from` and for converting them to named stages. build-config.env is now the single source of truth. No Dockerfile names a base directly: each takes it as an ARG whose default mirrors the config, so a plain `docker build` still works with no wrapper and CI can override any base with --build-arg. `scripts/ci/check-base-image-single-source.sh` fails on drift, on a pin that is not digest-pinned, and on a version knob that disagrees with its pin; `make base-images-sync` rewrites the mirrors. Same authority-plus-drift- check shape as check-default-model-single-source.sh. Removes four Debian 12 pins and two Ubuntu 24.04 CUDA pins. `vmafx-controller` gains an explicit `USER 65532:65532`: its old cc-debian12 pin lacked the `:nonroot` suffix, so it ran as root. Both its listen ports are above 1024. ROCm and oneAPI keep Ubuntu 24.04 under explicit, self-closing exemptions. Each is a major SDK migration needing matching source changes, not a pin swap: ROCm 7.2.4 -> 10.0.0 is PR #1386, and oneAPI 2025 -> 2026.1 is blocked on a libsycl soname bump (.so.8 -> .so.9), a glibc gap between Intel's Ubuntu images and Debian 13, and the NEO GPU driver that Intel's runtime image carries and Debian does not package. All measured; see the research digest. Also fixes a latent bug in dev/Containerfile: `ENV PYTHONPATH=/build/vmaf/python:${PYTHONPATH}` expanded to a trailing colon because PYTHONPATH was never set, putting the current directory on sys.path. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ovate The first commit unified container base images. That was half the problem: the same versions live in CI workflows, and they had drifted there independently. ROCm was 7.2.4 in build.yml and 7.2.3 in libvmaf-build-matrix.yml, so CI validated a ROCm the published images never shipped. The Level Zero loader existed at FOUR versions simultaneously -- v1.18.5, v1.28.0 twice, v1.29.0 and 1.32.0 -- and a skew there does not fail a build. It surfaces at runtime as "No device of requested type available", which reads like a driver or permissions problem and is not. build-config.env now carries the toolchain versions too (ROCM_APT_VERSION, LEVEL_ZERO_VERSION, CUDA_APT_PACKAGE, PYTHON_CI_VERSION). Workflow `run:` steps source it directly; scripts/ci/load-build-config.sh exports every knob into $GITHUB_ENV for steps that need a value earlier than that. The Windows SYCL leg runs under cmd and cannot source it, so it mirrors the value by hand and the gate keeps that mirror honest. scripts/ci/check-workflow-versions.py enforces it. It is a separate script because the Level Zero clone puts `--branch vX.Y.Z` and the repository URL on different lines, so a line-oriented grep silently sees only the first line -- the first version of this check passed against a deliberately planted literal. Renovate is reconfigured to match. Its `dockerfile` manager understands `ARG X=image` + `FROM $X`, so left alone it would bump the Dockerfile mirrors and leave build-config.env behind -- failing this change's own gate on Renovate's PRs. One custom manager now matches both the config line and the ARG form across all nine wired files (41 pins), so a bump lands every copy in one PR, and a packageRule disables the built-in manager on exactly those files. docker/dev/*.Dockerfile deliberately keeps the built-in manager. The gate also classifies config keys by the SHAPE of their value rather than by a list of names, so adding a version knob cannot accidentally subject it to the digest-pinning rule. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ADR-1225 landed ROCm 10.0.0 on the ubuntu-24.04 variant, which left the tree with the last release images on a distro everything else had moved off. AMD publishes rocm/dev-ubuntu-26.04:10.0.0-full too, but this is not a pin swap to make casually: Dockerfile.node and Dockerfile.production-gpu COPY these libraries out of the vendor image into a Debian 13 runtime, so a libc mismatch does not fail the build -- it fails the load, on the user's machine, with a "cannot open shared object file" that points at the wrong thing. Measured before moving: /opt/rocm/core-10.0/lib layout identical in both variants max GLIBC symbol in the closure 2.28 in both (AMD builds to an old floor) Debian 13 runtime provides 2.41 So the 26.04 variant is a drop-in. This is the OPPOSITE of the oneAPI case, where Intel's Ubuntu 26.04 image requires glibc 2.43 and genuinely cannot be copied onto Debian 13 -- the question has to be asked per vendor rather than answered once for all of them. With ROCm current, the gate's Ubuntu-24.04 exemption list is down to the two oneAPI entries, which disappear with the 2026.1 migration. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
build-config.env was added to stop version drift, but the oneAPI knob was
deliberately left out with a comment saying the 2026.1 migration was "the
immediate follow-up to this file". That follow-up never happened, so three
workflows kept spelling `intel-oneapi-compiler-dpcpp-cpp-2025.3` inline and the
CI toolchain sat two majors behind the compiler installed on the workstation.
All three already sourced build-config.env for LEVEL_ZERO_VERSION, so the
plumbing existed; only the version was missing from it. They now read
${ONEAPI_APT_PACKAGE}, and ONEAPI_VERSION is 2026.1.
The config also gains ONEAPI_RUNTIME_APT_PACKAGES, which names
intel-oneapi-umf explicitly: intel-oneapi-runtime-dpcpp-cpp does not depend on
it, so libumf.so.1 goes missing and the adapter fails to load -- surfacing as
"No device of requested type available" rather than as a link error. That was
measured while preparing the migration, and is the kind of fact the single
source exists to keep.
The Windows leg stays at 2025.3.0.372 on purpose. Its offline-installer URL
carries an opaque per-build GUID that cannot be derived from a version number,
so bumping it needs a looked-up URL rather than a string substitution; it is
recorded as ONEAPI_WINDOWS_VERSION so the lag is visible in the same file
instead of buried in a workflow, and tracked as
T-ONEAPI-WINDOWS-CI-LAG-2026-09-07.
Verified: load-build-config.sh resolves ONEAPI_APT_PACKAGE to
intel-oneapi-compiler-dpcpp-cpp-2026.1, no literal 2025.3 oneAPI package name
remains in .github/workflows/, and both check-workflow-versions.py and
check-base-image-single-source.sh pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…m-image Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(ffmpeg): replay percentile guard after the existing mappings * docs(adr): define stable-release FFmpeg patch maintenance * fix(ffmpeg): automate stable-release patch refresh and required replay * fix(go): confine predictor cards and require Go validation * docs(adr): preserve work during agent-state cleanup * fix(dev): preserve work during agent-state cleanup * fix(ci): recognize root build config in dependency PRs * fix(hooks): preserve dispatch across worktree removal * fix(ci): reject unpinned container image bypasses * fix(docs): refresh ADR metadata and enforce source freshness * fix(ci): align shared config consumers and combined documentation * fix(deps): restore Renovate custom manager file matching * fix(docs): require push toolchain and repair guide links * test(ffmpeg): type patch receipts and safety fixtures for push checks * test(ci): type impact and Go workflow contracts * docs: repair topic links and identify unavailable evidence * fix(build): consume the shared Level Zero version * docs: refresh FFmpeg guide and correct PSNR evidence * fix(dev): make container shell failures explicit * fix(hooks): retain documentation checks on large diffs --------- Co-authored-by: Lusoris <lusoris@pm.me>
lusoris
disabled auto-merge
September 8, 2026 16:20
lusoris
marked this pull request as draft
September 8, 2026 16:20
lusoris
force-pushed
the
build/base-image-single-source
branch
from
September 8, 2026 16:22
834f840 to
851d53d
Compare
This was referenced Sep 8, 2026
lusoris
marked this pull request as ready for review
September 15, 2026 20:05
…xcluding
The Dev Container build failed on this PR with
failed to compute cache key: "/build-config.env": not found
even though `build-config.env` sits at the repository root, is tracked, and no
`.dockerignore` pattern names it. Docker cleans a trailing slash off an ignore
pattern, so `build*/` is matched as `build*`, which also matches the new
root-level *file* this PR introduces, and the file never enters the build
context. `COPY build-config.env` then cannot find it.
Reproduced from first principles rather than inferred: a two-line
`.dockerignore` containing only `build*/` and `build-*/`, plus a Dockerfile whose
sole instruction is `COPY build-config.env`, fails with that exact message on
Docker 29.8.0. Adding `!build-config.env` makes it succeed, and a follow-up build
confirms `build-cpu/` is still excluded, so the directory patterns keep working.
This is the last thing standing between the single-source config and a green
container build — which matters beyond this PR, because master's `Docker Image
Build` has been red since 2026-09-08 on the FFmpeg patch series that this branch
also repairs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`Docker Image Build` fails at the first `apt-get install` with
E: Encountered a section with no Package: header
E: Problem with MergeList /var/lib/apt/lists/developer.download.nvidia.com_
compute_cuda_repos_ubuntu2604_x86%5f64_Packages.lz4
NVIDIA's ubuntu2604 CUDA index is currently malformed. Verified as upstream and
not ours by running the pinned digest directly — `docker run
nvidia/cuda:13.3.1-devel-ubuntu26.04@sha256:8cf42b8d… apt-get update` reproduces
it with no repository content involved — and the digest is byte-identical on
master and on this branch, so the base image did not change here.
None of the packages installed in that step come from the CUDA repo, and the
toolkit is baked into the image rather than installed from it: after removing the
list, `nvcc --version` still reports `cuda_13.3.r13.3`, and the install of
build-essential, ccache, ninja-build, nasm, python3 and pkg-config succeeds. The
whole first stage now builds locally to a clean export.
This is the same remedy `.github/workflows/go-ci.yml` already applies to the
Microsoft and Azure sources on hosted runners, for exactly this class of
third-party-index breakage.
Second of the two container failures on this branch. The first was the FFmpeg
patch series: patch 0018 stopped applying at `vf_libvmaf.c:940` once percentile
pooling landed twice, which is what has kept master's `Docker Image Build` and
`FFmpeg SYCL` red since 2026-09-08, and which the 0018 refresh already carried
here repairs — `ffmpeg_patch_stack.py --check` replays all 18 patches onto n9.0.1.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
lusoris
added a commit
that referenced
this pull request
Sep 15, 2026
…needs ADR-1248 created ruleset `VMAFx master security` with no bypass actors and one required independent approving review. `VMAFx` is an organisation with exactly one collaborator, `lusoris`, who authors every open pull request, and GitHub forbids approving your own pull request. The required approval therefore had nobody who could give it. Measured: the last merge in the repository was #1413 at 2026-09-08 16:15 UTC and the ruleset was created at 23:26 local the same day. Nothing merged in the week that followed — not #1396 at 70 passing checks, which carries the container fixes master's own `Docker Image Build` and `FFmpeg SYCL` lanes needed, not the security dependency bumps #1428 and #1429, not any of the pre-rc.1 correctness fixes. Every merge in the repository's history predates the ruleset and had zero reviews, so the control was never satisfied, only avoided by administrator override, which is the behaviour ADR-1248 set out to prevent and with no record of when it happened. The ruleset keeps every other control: one required approval, dismissal of stale reviews on push, approval after the last push, resolved review threads, strict up-to-date `Required Checks Aggregator`, linear history, blocked deletion and force pushes. It gains exactly one bypass actor, named by numeric user id rather than a role tier so it cannot silently widen when another admin is added. `scripts/dev/check_repository_security.py` changes from "the ruleset must have no bypass actors" to "the ruleset's bypass actors must be exactly the declared list". An undeclared actor, a second actor beside the declared one, and a widening from `always` to `exempt` all still fail. The bypass waives the approval, never the tests. The honest limit, recorded in the ADR, the operator page and the script: a reader without ruleset write access sees only `bypassActors.totalCount`, so it can verify how many actors bypass but not which. A reader with admin rights compares identities exactly, and the Scorecard master run does. Remove the bypass and supersede ADR-1252 when a second maintainer can review. ADR-1252 supersedes ADR-1248. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
6 of 8 tasks
lusoris
added a commit
that referenced
this pull request
Sep 15, 2026
…needs ADR-1248 created ruleset `VMAFx master security` with no bypass actors and one required independent approving review. `VMAFx` is an organisation with exactly one collaborator, `lusoris`, who authors every open pull request, and GitHub forbids approving your own pull request. The required approval therefore had nobody who could give it. Measured: the last merge in the repository was #1413 at 2026-09-08 16:15 UTC and the ruleset was created at 23:26 local the same day. Nothing merged in the week that followed — not #1396 at 70 passing checks, which carries the container fixes master's own `Docker Image Build` and `FFmpeg SYCL` lanes needed, not the security dependency bumps #1428 and #1429, not any of the pre-rc.1 correctness fixes. Every merge in the repository's history predates the ruleset and had zero reviews, so the control was never satisfied, only avoided by administrator override, which is the behaviour ADR-1248 set out to prevent and with no record of when it happened. The ruleset keeps every other control: one required approval, dismissal of stale reviews on push, approval after the last push, resolved review threads, strict up-to-date `Required Checks Aggregator`, linear history, blocked deletion and force pushes. It gains exactly one bypass actor, named by numeric user id rather than a role tier so it cannot silently widen when another admin is added. `scripts/dev/check_repository_security.py` changes from "the ruleset must have no bypass actors" to "the ruleset's bypass actors must be exactly the declared list". An undeclared actor, a second actor beside the declared one, and a widening from `always` to `exempt` all still fail. The bypass waives the approval, never the tests. The honest limit, recorded in the ADR, the operator page and the script: a reader without ruleset write access sees only `bypassActors.totalCount`, so it can verify how many actors bypass but not which. A reader with admin rights compares identities exactly, and the Scorecard master run does. Remove the bypass and supersede ADR-1252 when a second maintainer can review. ADR-1252 supersedes ADR-1248. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
lusoris
added a commit
that referenced
this pull request
Sep 15, 2026
…needs ADR-1248 created ruleset `VMAFx master security` with no bypass actors and one required independent approving review. `VMAFx` is an organisation with exactly one collaborator, `lusoris`, who authors every open pull request, and GitHub forbids approving your own pull request. The required approval therefore had nobody who could give it. Measured: the last merge in the repository was #1413 at 2026-09-08 16:15 UTC and the ruleset was created at 23:26 local the same day. Nothing merged in the week that followed — not #1396 at 70 passing checks, which carries the container fixes master's own `Docker Image Build` and `FFmpeg SYCL` lanes needed, not the security dependency bumps #1428 and #1429, not any of the pre-rc.1 correctness fixes. Every merge in the repository's history predates the ruleset and had zero reviews, so the control was never satisfied, only avoided by administrator override, which is the behaviour ADR-1248 set out to prevent and with no record of when it happened. The ruleset keeps every other control: one required approval, dismissal of stale reviews on push, approval after the last push, resolved review threads, strict up-to-date `Required Checks Aggregator`, linear history, blocked deletion and force pushes. It gains exactly one bypass actor, named by numeric user id rather than a role tier so it cannot silently widen when another admin is added. `scripts/dev/check_repository_security.py` changes from "the ruleset must have no bypass actors" to "the ruleset's bypass actors must be exactly the declared list". An undeclared actor, a second actor beside the declared one, and a widening from `always` to `exempt` all still fail. The bypass waives the approval, never the tests. The honest limit, recorded in the ADR, the operator page and the script: a reader without ruleset write access sees only `bypassActors.totalCount`, so it can verify how many actors bypass but not which. A reader with admin rights compares identities exactly, and the Scorecard master run does. Remove the bypass and supersede ADR-1252 when a second maintainer can review. ADR-1252 supersedes ADR-1248. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Container base images were pinned by hand at every point of use, and the copies
drifted.
Dockerfile.controllerandDockerfile.operatorstill built onDebian 12 (
golang:1.27-bookworm) and shipped ondistroless/cc-debian12/distroless/static-debian12long after the rest of the tree moved to Debian 13— and both quoted a golang digest in their own header comments (
ded31c68…)that did not match the digest in their own
FROMline (648f440f…). Two CUDApins sat on Ubuntu 24.04 beside a sibling stage already on 26.04.
Four more pins were not
FROMlines at all.dev/Containerfilelifted theGo toolchain out of a
golang:1.27-bookwormimage withCOPY --from=<image>,and
docker/Dockerfile.nodepulled the CUDA, ROCm and oneAPI runtime librariesthe same way. Those decide the libc and SDK the shipped artifact carries, but
they are invisible to anyone grepping for
FROM— which is exactly why theywere the most out-of-date images in the repository.
build-config.envis now the single source of truth. No Dockerfile names abase directly: each takes it as an
ARGwhose default mirrors the config, so aplain
docker buildstill works with no wrapper and CI can override any basewith
--build-arg.COPY --frompins became named stages.scripts/ci/check-base-image-single-source.shfails on drift, on anon-digest-pinned entry, and on a version knob that disagrees with its pin;
make base-images-syncrewrites the mirrors. Same authority-plus-drift-checkshape as
check-default-model-single-source.sh.Net: 25
FROMlines across 10 Dockerfiles collapse to 13 pins in one file.Four Debian 12 pins and two Ubuntu 24.04 CUDA pins are gone; no live Debian 12
reference remains.
Behaviour change worth a reviewer's eye
vmafx-controllernow runs as UID 65532 instead of root. Its old pin wasgcr.io/distroless/cc-debian12without the:nonrootsuffix; the sharedruntime pin is
:nonroot. Rather than let the tag decide it silently, theUSER 65532:65532line is written explicitly in the file. Both listen ports(8080, 9090) are above 1024, so nothing needs a capability.
Why oneAPI and ROCm are still on 24.04
Both keep an explicit, self-closing exemption in the gate. Neither is a pin
swap:
ROCm 7.2.4 → 10.0.0 is feat(hip): migrate every ROCm consumer to 10.0.0 from digest-pinned images (ADR-1225) #1386, which carries the matching HIP changes.
Bumping a major GPU SDK here without them would be reckless. Target is
rocm/dev-ubuntu-26.04:10.0.0-full.oneAPI 2025 → 2026.1 is a restructure. Measured, not assumed:
intel/oneapi-basekitis a retired repository (last tag 2025.3.2).Intel continued under the plain name
intel/oneapi, now at 2026.1.0.Auditing the old name makes the toolchain look two years stale.
DT_NEEDED libsycl.so.8, 2026.xships
libsycl.so.9. A major soname bump is the ABI break, so both sidesmust cross together — the usual runtime ≥ compiler rule only holds while the
soname is stable.
intel/oneapiis at2026.1.0 but
intel/oneapi-runtimestops at 2026.0.0 (runtime behindcompiler), and both are Ubuntu-only — Ubuntu 26.04 is glibc 2.43, Debian
13 is glibc 2.41, so a binary built in Intel's image cannot load on the
Debian 13 runtime the release track ships.
compiler and runtime both at 2026.1.1-325,
libsycl.so.9resolving.(
libze_intel_gpu.so.1, OpenCL ICDs) which Debian does not package, so anaive move would ship a GPU image that can't see the GPU. The fork already
solves this for dev via
dev/scripts/fetch-intel-neo.py(ADR-1145).Folding that into this PR would have shipped a broken GPU image. It is the
immediate follow-up; everything needed is in the digest.
Type
build/ci— tooling / infraChecklist
make format && make lintis green locally.meson test -C build. — no C/C++ code touched; this PR changes container definitions, docs and a shell gate only./cross-backend-diffand the worst ULP is ≤ 2. — no SIMD/GPU code path touched..c/.cpp/.cu/.h/.hpp, it has the appropriate license header (seeCONTRIBUTING.md). — no new C/C++ sources.!orBREAKING CHANGE:and the migration path is documented below. — not a breaking change.docs/adr/_index_fragments/<NNNN-slug>.mdand the slug is appended todocs/adr/_index_fragments/_order.txt.Bug-status hygiene (ADR-0165)
docs/state.mdupdated in this PR with a row —T-DOCKER-BASE-IMAGE-DRIFT-2026-09-07under Recently closed.Netflix golden-data gate (ADR-0024)
assertAlmostEqual(...)score in the Netflix golden Python tests.Deep-dive deliverables (ADR-0108)
docs/research/1231-base-image-single-source.md: the drift inventory, the oneAPI soname/glibc/NEO measurements, and reproduction commands.## Alternatives considered(config+ARG+gate vs. no-default ARGs vs.buildx bakevs. Renovate-only).AGENTS.mdinvariant note — added to bothdocker/AGENTS.mdanddev/AGENTS.md.changelog.d/changed/base-image-single-source.md(CHANGELOG.mdregenerated viascripts/release/concat-changelog-fragments.sh --write).docs/rebase-notes.md, sectionADR-1231 — container base images come from build-config.env (2026-09-07).Reproducer
Known follow-ups
fetch-intel-neo.py; renames the-oneapi2025stage/tag. Removes theoneAPI half of the gate exemption.
build-config.envplus its HIP changes, rather than four Dockerfile edits.
🤖 Generated with Claude Code