Skip to content

fix(build): resolve Windows CUDA compiler fallback and clarify runtime ownership - #1421

Closed
lusoris wants to merge 10 commits into
masterfrom
fix/rc1-hygiene-followup
Closed

lusoris wants to merge 10 commits into
masterfrom
fix/rc1-hygiene-followup

Conversation

@lusoris

@lusoris lusoris commented Sep 8, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Windows CUDA configuration failed with Unknown variable cl_path when vswhere was unavailable or returned no compiler, even if cl.exe was on PATH. Assign the fallback path once and use it for both NVCC and MSVC header discovery. A Meson configure regression exercises the shipped discovery block and runs in the POSIX fast suite.

Also correct native-versus-Python ONNX Runtime ownership documentation: the native CPU archive does not provide CUDA or ROCm execution providers. Runtime pins are unchanged.

This is a held follow-up to #1396. The preceding hygiene PR #1420 was merged into that parent by the old local train, not into master. The parent and this follow-up remain drafts pending full integration acceptance; stacked PRs must not be auto-promoted or merged by the train.

Validation

  • Four configure cases pass with Meson 1.10.1 and 1.12.0; both fallback cases fail against the original source with the undefined-variable error.
  • Actual Meson --suite=fast registration executes the fixture successfully (1/1 registered test, four internal cases).
  • Ruff lint/format and normal pre-commit/commit-message gates pass. Generated changelog freshness is checked.
  • The native Windows CI sample linked in Netflix/vmaf#1472 proves compile/link/install on its recorded earlier revision. This configure regression does not establish Windows GPU runtime or numerical parity.
  • Full combined make lint, make test, fresh image, golden-data and hosted acceptance remain pending. No merge-readiness claim is made from focused checks.

Bug-status hygiene

  • docs/state.md records the Windows fallback bug and the ORT ownership correction.
  • Netflix golden-score assertions are unchanged.

Deep-dive deliverables

  • Research digest — no digest needed: trivial compiler-path assignment and documentation correction, with executable negative controls.
  • Decision matrix — no alternatives: only-one-way fix; both consumers require the compiler path selected by the existing fallback.
  • AGENTS.md invariant note — core/src/AGENTS.md records the shared compiler-path contract.
  • Reproducer / smoke-test command — below.
  • CHANGELOG fragment — changelog.d/fixed/windows-cuda-cl-path-fallback.md and changelog.d/fixed/ort-runtime-owner-comments.md.
  • Rebase note — docs/rebase-notes.md records both contracts.

Reproducer

python3 core/test/test_windows_cuda_compiler_discovery.py -v

Native compilation and hardware scoring remain separate validation steps.

Verified exact-source validation update — 2026-09-08

This supersedes the earlier statement that make test remained pending. Current PR head cc022245079f1b7da4005a39bcefc759e2d6ef3d (tree c2d83fb0798f4989874ef550e60de31f69a832a8) passed the configured CPU make test command: exit 0, 134/134 Meson tests passed, with ASM and default LTO enabled. The source was mounted read-only in an isolated offline container; the installed application binary was not used.

Full make -k lint exited 2 because clang-tidy, black, shellcheck, npx, and gosec were missing. cppcheck was also absent, but lint-c stopped before reaching it. Ruff's configured scope and generated-document consistency checks passed; that does not make full lint green. Exact commands, logs, configuration, hashes, and limitations are retained in .workingdir2/evidence/2026-09-08-rc1-full-gates-cc022245/README.md. The commands select verified Meson/Ninja paths and suppress only their offline bootstrap attempts; source compilation, tests, and analyzer failures are preserved.

The Makefile's separate Python golden-data and sanitizer targets were not run. CUDA, SYCL, HIP, Metal, DNN, MCP, and docs generation were disabled. This GCC 15 CPU result does not establish backend, native Windows/macOS, canonical GCC 14 CI, or RC1 acceptance. Full lint and the remaining integration gates are still required; this PR remains draft and held.

Ancestry is verified: cc02224 → a49d54f → current draft parent #1396 at 851d53dc8112e103e5f55544c09b3dc78f02f7da. Thus the CPU receipt applies to this current combined source without a new rebase. The original #1420 merge event produced 834f84063a4640e72b0916a9a5b3a566b6810fec on that parent branch only; the legacy train subsequently rebased it onto master 78c9d2bf at 18:17:41 +0200, before suspension. The hygiene stack was not merged into master.

Related held work: #1422 contains the queue/backpressure repair; #1423 contains guarded train control and its paused migration helper. #1416 configuration convergence remained pending push at this update (remote head d71e4336). Runtime installation belongs to the coordinating operator; this documentation update changes no holds, draft states, or processes.

Canonical CPU golden follow-up — 2026-09-08, exact cc02224

The unchanged original QualityRunnerTest::test_run_vmaf_runner and
QualityRunnerTest::test_run_vmaf_runner_checkerboard nodes both passed
(2/2). Together they cover all three canonical pairs: normal video and
both checkerboard shifts, plus self-reference and feature-score checks.
No original assertion or tolerance changed. Only executable/resource
paths were bound to the exact retained CPU binary and input data; the same
cc02224 source was mounted read-only with a fresh output workspace.

The bounded CPU container run used NumPy distribution/module 2.5.3 and
the hash-verified official PyWavelets 1.10.0 distribution, installed only
in a private read-only overlay. The PyWavelets wheel's packaged module
reports 1.8.0, despite its distribution METADATA reporting 1.10.0. The
two runtime floors are therefore resolved at distribution metadata level;
PyWavelets runtime version identity remains inconsistent. This limitation
is recorded without changing the package, source or golden assertions.

The central local receipt
is retained on the workstation at
.workingdir2/evidence/2026-09-08-rc1-canonical-golden-cc022245/floor-overlay/README.md
(not a hosted GitHub artifact). It includes official package metadata,
wheel/overlay/source/binary/data hashes, exact command and versions, the
initial version-identity preflight refusal, final passing JUnit/logs, and
unchanged-input verification. The initial passing run with existing
dependencies is retained separately in the parent evidence directory.

This supersedes only the earlier pending canonical three-pair CPU golden
status. The complete five-module make test-netflix-golden target, full lint,
sanitisers, other backends/platforms and hosted integration acceptance remain
separate and unverified by this run. This PR remains draft and held; no RC1
readiness or merge approval is claimed.

lusoris and others added 10 commits September 8, 2026 18:17
…ADR-1231)

Container base images were pinned by hand at each point of use and drifted.
`Dockerfile.controller` and `Dockerfile.operator` still built on Debian 12 and
shipped on distroless-debian12 after the rest of the tree moved to Debian 13,
and both quoted a golang digest in their header comments that did not match
their own FROM line. Two CUDA pins sat on Ubuntu 24.04 beside a sibling stage
on 26.04.

Four further pins were invisible to any FROM scan because they lived in
`COPY --from=<image>` -- the Go toolchain in dev/Containerfile and the CUDA,
ROCm and oneAPI runtime libraries in Dockerfile.node. Those were the most stale
images in the repository, which is the argument both for gating `COPY --from`
and for converting them to named stages.

build-config.env is now the single source of truth. No Dockerfile names a base
directly: each takes it as an ARG whose default mirrors the config, so a plain
`docker build` still works with no wrapper and CI can override any base with
--build-arg. `scripts/ci/check-base-image-single-source.sh` fails on drift, on
a pin that is not digest-pinned, and on a version knob that disagrees with its
pin; `make base-images-sync` rewrites the mirrors. Same authority-plus-drift-
check shape as check-default-model-single-source.sh.

Removes four Debian 12 pins and two Ubuntu 24.04 CUDA pins. `vmafx-controller`
gains an explicit `USER 65532:65532`: its old cc-debian12 pin lacked the
`:nonroot` suffix, so it ran as root. Both its listen ports are above 1024.

ROCm and oneAPI keep Ubuntu 24.04 under explicit, self-closing exemptions.
Each is a major SDK migration needing matching source changes, not a pin swap:
ROCm 7.2.4 -> 10.0.0 is PR #1386, and oneAPI 2025 -> 2026.1 is blocked on a
libsycl soname bump (.so.8 -> .so.9), a glibc gap between Intel's Ubuntu images
and Debian 13, and the NEO GPU driver that Intel's runtime image carries and
Debian does not package. All measured; see the research digest.

Also fixes a latent bug in dev/Containerfile: `ENV PYTHONPATH=/build/vmaf/python:${PYTHONPATH}`
expanded to a trailing colon because PYTHONPATH was never set, putting the
current directory on sys.path.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ovate

The first commit unified container base images. That was half the problem: the
same versions live in CI workflows, and they had drifted there independently.

ROCm was 7.2.4 in build.yml and 7.2.3 in libvmaf-build-matrix.yml, so CI
validated a ROCm the published images never shipped. The Level Zero loader
existed at FOUR versions simultaneously -- v1.18.5, v1.28.0 twice, v1.29.0 and
1.32.0 -- and a skew there does not fail a build. It surfaces at runtime as
"No device of requested type available", which reads like a driver or
permissions problem and is not.

build-config.env now carries the toolchain versions too (ROCM_APT_VERSION,
LEVEL_ZERO_VERSION, CUDA_APT_PACKAGE, PYTHON_CI_VERSION). Workflow `run:` steps
source it directly; scripts/ci/load-build-config.sh exports every knob into
$GITHUB_ENV for steps that need a value earlier than that. The Windows SYCL leg
runs under cmd and cannot source it, so it mirrors the value by hand and the
gate keeps that mirror honest.

scripts/ci/check-workflow-versions.py enforces it. It is a separate script
because the Level Zero clone puts `--branch vX.Y.Z` and the repository URL on
different lines, so a line-oriented grep silently sees only the first line --
the first version of this check passed against a deliberately planted literal.

Renovate is reconfigured to match. Its `dockerfile` manager understands
`ARG X=image` + `FROM $X`, so left alone it would bump the Dockerfile mirrors
and leave build-config.env behind -- failing this change's own gate on
Renovate's PRs. One custom manager now matches both the config line and the ARG
form across all nine wired files (41 pins), so a bump lands every copy in one
PR, and a packageRule disables the built-in manager on exactly those files.
docker/dev/*.Dockerfile deliberately keeps the built-in manager.

The gate also classifies config keys by the SHAPE of their value rather than by
a list of names, so adding a version knob cannot accidentally subject it to the
digest-pinning rule.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ADR-1225 landed ROCm 10.0.0 on the ubuntu-24.04 variant, which left the tree
with the last release images on a distro everything else had moved off.

AMD publishes rocm/dev-ubuntu-26.04:10.0.0-full too, but this is not a pin swap
to make casually: Dockerfile.node and Dockerfile.production-gpu COPY these
libraries out of the vendor image into a Debian 13 runtime, so a libc mismatch
does not fail the build -- it fails the load, on the user's machine, with a
"cannot open shared object file" that points at the wrong thing.

Measured before moving:

  /opt/rocm/core-10.0/lib layout   identical in both variants
  max GLIBC symbol in the closure  2.28 in both (AMD builds to an old floor)
  Debian 13 runtime provides       2.41

So the 26.04 variant is a drop-in. This is the OPPOSITE of the oneAPI case,
where Intel's Ubuntu 26.04 image requires glibc 2.43 and genuinely cannot be
copied onto Debian 13 -- the question has to be asked per vendor rather than
answered once for all of them.

With ROCm current, the gate's Ubuntu-24.04 exemption list is down to the two
oneAPI entries, which disappear with the 2026.1 migration.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
build-config.env was added to stop version drift, but the oneAPI knob was
deliberately left out with a comment saying the 2026.1 migration was "the
immediate follow-up to this file". That follow-up never happened, so three
workflows kept spelling `intel-oneapi-compiler-dpcpp-cpp-2025.3` inline and the
CI toolchain sat two majors behind the compiler installed on the workstation.

All three already sourced build-config.env for LEVEL_ZERO_VERSION, so the
plumbing existed; only the version was missing from it. They now read
${ONEAPI_APT_PACKAGE}, and ONEAPI_VERSION is 2026.1.

The config also gains ONEAPI_RUNTIME_APT_PACKAGES, which names
intel-oneapi-umf explicitly: intel-oneapi-runtime-dpcpp-cpp does not depend on
it, so libumf.so.1 goes missing and the adapter fails to load -- surfacing as
"No device of requested type available" rather than as a link error. That was
measured while preparing the migration, and is the kind of fact the single
source exists to keep.

The Windows leg stays at 2025.3.0.372 on purpose. Its offline-installer URL
carries an opaque per-build GUID that cannot be derived from a version number,
so bumping it needs a looked-up URL rather than a string substitution; it is
recorded as ONEAPI_WINDOWS_VERSION so the lag is visible in the same file
instead of buried in a workflow, and tracked as
T-ONEAPI-WINDOWS-CI-LAG-2026-09-07.

Verified: load-build-config.sh resolves ONEAPI_APT_PACKAGE to
intel-oneapi-compiler-dpcpp-cpp-2026.1, no literal 2025.3 oneAPI package name
remains in .github/workflows/, and both check-workflow-versions.py and
check-base-image-single-source.sh pass.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…m-image

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(ffmpeg): replay percentile guard after the existing mappings

* docs(adr): define stable-release FFmpeg patch maintenance

* fix(ffmpeg): automate stable-release patch refresh and required replay

* fix(go): confine predictor cards and require Go validation

* docs(adr): preserve work during agent-state cleanup

* fix(dev): preserve work during agent-state cleanup

* fix(ci): recognize root build config in dependency PRs

* fix(hooks): preserve dispatch across worktree removal

* fix(ci): reject unpinned container image bypasses

* fix(docs): refresh ADR metadata and enforce source freshness

* fix(ci): align shared config consumers and combined documentation

* fix(deps): restore Renovate custom manager file matching

* fix(docs): require push toolchain and repair guide links

* test(ffmpeg): type patch receipts and safety fixtures for push checks

* test(ci): type impact and Go workflow contracts

* docs: repair topic links and identify unavailable evidence

* fix(build): consume the shared Level Zero version

* docs: refresh FFmpeg guide and correct PSNR evidence

* fix(dev): make container shell failures explicit

* fix(hooks): retain documentation checks on large diffs

---------

Co-authored-by: Lusoris <lusoris@pm.me>
Base automatically changed from build/base-image-single-source to master September 15, 2026 21:08
@lusoris

lusoris commented Sep 15, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by the pre-rc.1 fixing train in #1425.

Verified by comparing this branch's tree against the train for every path its commits touch: zero differences. Its own commits are on the train already, and everything else it carried was the base-image chain that #1396's squash put on master on 2026-09-15.

Closing is safe; nothing here is lost. If you would rather keep it open until #1425 merges, that works too — it just has nothing left to contribute.

@lusoris

lusoris commented Sep 15, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by the pre-rc.1 fixing train in #1425.

Closing because the ADR Collision Guard is a required check on #1425 and fails while two open PRs claim the same ADR numbers. Verified before closing:

The branch is untouched, so this is reversible.

@lusoris lusoris closed this Sep 15, 2026
@lusoris
lusoris deleted the fix/rc1-hygiene-followup branch September 18, 2026 07:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant