Skip to content

fix(docker): run the oneAPI release image on Debian 13 with the pinned Intel GPU runtime - #1629

Merged
lusoris merged 4 commits into
masterfrom
fix/release-oneapi-image-runtime
Sep 30, 2026
Merged

lusoris merged 4 commits into
masterfrom
fix/release-oneapi-image-runtime

Conversation

@lusoris

@lusoris lusoris commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

Summary

The published oneAPI image crashed every vmaf --backend sycl run on an Arc B580 (exit 139 right after device selection); a UHD 770 worked. The cause is the Intel GPU compute runtime (NEO 25.18) that Intel's oneapi-runtime:2025.3.1 image carries. In the unchanged image, swapping only that runtime for NEO 26.35 stops the crash, and swapping only the Level Zero loader does not.

This PR moves the image to the design build-config.env already described (ADR-1368):

  • Builder and final stage start from the release track's debian:13-slim, like the CPU image.
  • Intel's oneAPI 2026.1 compiler (builder) and SYCL runtime plus UMF (final stage) come from Intel's apt repository at one exact build (2026.1.1-325), through the new scripts/ci/install-intel-oneapi.sh. The repository key is pinned by fingerprint.
  • The NEO GPU runtime at INTEL_NEO_VERSION (the package set the dev container installs) and the Level Zero loader at LEVEL_ZERO_VERSION come from the existing scripts/ci/install-intel-ocloc.sh, which gains --components build|runtime. The fetcher checks the loader against GitHub's asset digests.
  • The image is published as -oneapi2026. The -oneapi2025 tag and the final-oneapi2025 stage stay as aliases of the same image (HISS-14).
  • The base-image gate now requires ONEAPI_BUILDER and ONEAPI_RUNTIME to equal RELEASE_BUILDER_BASE, and its distro exemption list is empty.

It also fixes a second break found on the way: on current master the 2025.3.2 builder no longer links the unit tests. libsycl-devicelib-host.a needs libm after --as-needed has dropped it. The 2026.1.1 compiler links them.

Closes T-RELEASE-ONEAPI-IMAGE-B580-SIGSEGV-2026-09-29.

Type

  • fix — bug fix
  • build / ci — tooling / infra

Checklist

  • Commits follow Conventional Commits (the commit-msg hook enforces this).
  • make format && make lint is green locally. The repository's pre-commit framework passes on this diff in a Linux container (every hook, REUSE included), plus hadolint 2.14.0, shellcheck 0.11.0, shfmt 3.13.1 and actionlint. praetorctl governance was not available on this host; CI runs it.
  • Unit tests pass: python3 scripts/ci/run_meson_test.py -- -C build. Not run separately; the image builder compiles and links every unit test with the 2026.1.1 compiler. The changed Python and shell contracts pass (fetcher 21 tests, base-image single-source 77, dev-container secret 31, CUDA installer 18, image runtime and publish source-binding contracts).
  • If I touched any SIMD/GPU code path, I ran /cross-backend-diff and the worst ULP is ≤ 2. No kernel code changed. The image's SYCL scores were compared with its CPU backend under the parity-gate tolerances (table below).
  • If I touched a feature extractor with SIMD/GPU twins, I either updated every twin or listed the gap under "Known follow-ups" below. No extractor touched.
  • If I added a new .c / .cpp / .cu / .h / .hpp, it has the appropriate license header (see CONTRIBUTING.md). None added; the new shell script carries the EUPL-1.2 header.
  • If this is a breaking change, the commit message uses ! or BREAKING CHANGE: and the migration path is documented below. Not breaking: both old names keep working.
  • If this PR adds an ADR, the ADR row lives in docs/adr/_index_fragments/<NNNN-slug>.md and the slug is appended to docs/adr/_index_fragments/_order.txt.

Bug-status hygiene (ADR-0165)

  • docs/state.md updated in this PR: T-RELEASE-ONEAPI-IMAGE-B580-SIGSEGV-2026-09-29 moved to "Recently closed", with the evidence and a copy-paste check for a native Linux host.

Netflix golden-data gate (ADR-0024)

  • I did not modify any assertAlmostEqual(...) score in the Netflix golden Python tests.
  • If I believe a golden value must change, I have explained why below AND pinged @lusoris for a CODEOWNERS exception. No golden value changes.

Cross-backend numerical results

--backend sycl in the rebuilt image against --backend cpu in the same image. Windows 11 host, Docker Desktop WSL 2 backend, /dev/dxg. Every GPU run held flock /f/gpu.lock. Tolerances are the parity gate's (FEATURE_TOLERANCE, metric_delta, area_tolerance_factor; the VMAF score at places=4), checked per frame and per metric. Each JSON's feature_backends confirms the SYCL twin ran.

Image built at this branch's head (final-oneapi2026, AOT image check: 30 fat binaries, 19 targets).

Pair Workload Device CPU pooled SYCL pooled Worst per-frame delta (tolerance) Result
Netflix, 48 frames default model (vmaf) Arc B580 82.816059 82.816058 integer_motion2 1.2e-5 (5e-5) pass
Netflix, 48 frames default model (vmaf) UHD 770 82.816059 82.816058 integer_motion2 1.2e-5 (5e-5) pass
Netflix, 48 frames --feature psnr_hvs Arc B580 31.330446 31.330390 psnr_hvs_y 8.3e-5 (5e-4) pass
Netflix, 48 frames --feature psnr_hvs UHD 770 31.330446 31.330390 psnr_hvs_y 8.3e-5 (5e-4) pass
Netflix, 48 frames --feature ssimulacra2 Arc B580 24.614428 24.614428 0 on every frame (5e-3) pass
Netflix, 48 frames --feature ssimulacra2 UHD 770 24.614428 24.614428 0 on every frame (5e-3) pass
BBB 4K, 50 frames default model (vmaf) Arc B580 79.182228 79.182227 integer_motion2 5e-6 (5e-5) pass
BBB 4K, 50 frames default model (vmaf) UHD 770 79.182228 79.182227 integer_motion2 5e-6 (5e-5) pass
BBB 4K, 50 frames --feature psnr_hvs Arc B580 44.258029 44.257588 psnr_hvs_y 8.43e-4 (3.34e-3, area-scaled) pass
BBB 4K, 50 frames --feature psnr_hvs UHD 770 44.258029 44.257588 psnr_hvs_y 8.43e-4 (3.34e-3, area-scaled) pass
BBB 4K, 50 frames --feature ssimulacra2 Arc B580 65.881933 65.881933 0 on every frame (5e-3) pass
BBB 4K, 50 frames --feature ssimulacra2 UHD 770 65.881933 65.881933 0 on every frame (5e-3) pass

Root-cause probe

The same 2025-built binary (final-oneapi2025 at 2d9d5b069) in every row, with packages swapped. Default model, Netflix pair, 2 frames.

Variant NEO IGC Level Zero loader Arc B580 UHD 770
image as published 25.18.33578.15 2.11.12 1.21.9 exit 139 exit 0
loader only 25.18.33578.15 2.11.12 1.34.0 exit 139 exit 0
NEO only 26.35.39758.10 2.41.5 1.21.9 exit 0 exit 0
NEO + loader (dev container set) 26.35.39758.10 2.41.5 1.34.0 exit 0 exit 0

Image size

before (final-oneapi2025, probe at 2d9d5b069) after (final-oneapi2026)
Unpacked 5.86 GB 2.39 GB
docker save + gzip -6 1.42 GB 0.61 GB

Deep-dive deliverables (ADR-0108)

  • Research digest — docs/research/2128-oneapi-release-image-runtime.md: the probe, Intel's apt repository layout and pins, the key, the rebuilt image and its scores.
  • Decision matrix — in ADR-1368 ## Alternatives considered: overlay NEO on the 2025 images, Intel's 2026 images, Debian 13 with pinned packages (chosen), Debian's Level Zero loader, a second NEO installer, switching dev/Containerfile now, and four tag-naming options.
  • AGENTS.md invariant note — docker/AGENTS.md (oneAPI bases and the image's load-bearing steps) and dev/AGENTS.md (the shared installer and fetcher).
  • Reproducer / smoke-test command — below.
  • CHANGELOG fragment — changelog.d/fixed/release-oneapi-image-runtime.md and changelog.d/changed/release-oneapi-image-tag.md.
  • Rebase note — docs/rebase-notes.md, "ADR-1368 — oneAPI release image on Debian 13 with pinned Intel packages".

Reproducer

On a Linux host with an Intel GPU:

docker build -f docker/Dockerfile.production-gpu --target final-oneapi2026 -t vmafx:oneapi2026-check .
for b in cpu sycl; do
  docker run --rm --device /dev/dri --group-add "$(getent group render | cut -d: -f3)" \
    -e ONEAPI_DEVICE_SELECTOR=level_zero:gpu -v "$PWD/testdata:/t:ro" vmafx:oneapi2026-check \
    --backend "$b" --reference /t/ref_576x324_48f.yuv --distorted /t/dis_576x324_48f.yuv \
    --width 576 --height 324 --pixel_format 420 --bitdepth 8 --json --output /dev/stdout \
    | jq -c '[.backend_used, .pooled_metrics.vmaf.mean]'
done

On Windows with Docker Desktop, pass --device /dev/dxg -v /usr/lib/wsl:/usr/lib/wsl:ro instead of /dev/dri, and append :/usr/lib/wsl/lib to the image's LD_LIBRARY_PATH (see docs/development/docker-production.md).

Known follow-ups

  • Not verified on a native Linux host or on an Arc A380. WSL 2 reaches the GPU through /dev/dxg and the host driver's /usr/lib/wsl/lib, which is not the i915/xe render-node path. The state row carries the copy-paste check for the maintainer's RTX 4090 + Arc A380 box.
  • dev/Containerfile keeps its own NEO and Level Zero steps. They install the same package set through the same fetcher, but their RUNs are bound to the BuildKit-secret contract (ADR-1271) and check-workflow-versions.py, so moving them onto the installer is a separate change.
  • The CI SYCL legs still add Intel's apt repository inline and build the Level Zero loader from source. install-intel-oneapi.sh and --components build could replace both.
  • A recovery dispatch (ADR-1347) that builds a pre-rc.3 tag with this recipe cannot build the oneAPI image, because the old tags lack the installers and knobs. This was already true since ADR-1360 added install-intel-ocloc.sh.
  • node-sycl (unpublished) builds libvmaf without SYCL, so the runtime it copies is unused. Unchanged here; the target builds with the new runtime stage.

🤖 Generated with Claude Code

@lusoris
lusoris force-pushed the fix/release-oneapi-image-runtime branch from c9f2e7b to a74842e Compare September 29, 2026 14:36
@github-actions github-actions Bot added the type:bug Something isn't working label Sep 29, 2026
Comment thread build-config.env Fixed
@lusoris
lusoris force-pushed the fix/release-oneapi-image-runtime branch from a74842e to 9675aa6 Compare September 29, 2026 14:47
@lusoris
lusoris force-pushed the fix/release-oneapi-image-runtime branch from 9675aa6 to b9cc797 Compare September 29, 2026 16:52
lusoris and others added 3 commits September 30, 2026 10:21
…d Intel GPU runtime

The published oneAPI image crashed every `vmaf --backend sycl` run on an
Arc B580 (exit 139 right after device selection). The cause is the Intel
GPU compute runtime (NEO 25.18) that Intel's oneapi-runtime:2025.3.1 image
carries: swapping only that runtime for NEO 26.35 in the unchanged image
fixes the crash, and swapping only the Level Zero loader does not.

The image now follows the design build-config.env already recorded
(ADR-1368). Builder and final stage start from the release track's
debian:13-slim. scripts/ci/install-intel-oneapi.sh installs Intel's oneAPI
2026.1 compiler or SYCL runtime plus UMF at apt build 2026.1.1-325, with
the repository key pinned by fingerprint. install-intel-ocloc.sh gains
`--components build|runtime`, which adds the NEO GPU runtime at
INTEL_NEO_VERSION (the package set the dev container uses) and the Level
Zero loader at LEVEL_ZERO_VERSION, checked against GitHub's asset digests.
The base-image gate now requires ONEAPI_BUILDER and ONEAPI_RUNTIME to equal
RELEASE_BUILDER_BASE, and its distro exemption list is empty.

The image is published as `-oneapi2026`. The `-oneapi2025` tag and the
`final-oneapi2025` stage stay as aliases of the same image.

In the rebuilt image the B580 and a UHD 770 match `--backend cpu` within
the parity gate for the default model, psnr_hvs and ssimulacra2 on the
Netflix pair and BBB 4K. The image shrinks from 5.86 GB to 2.39 GB.

Closes T-RELEASE-ONEAPI-IMAGE-B580-SIGSEGV-2026-09-29.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lusoris
lusoris force-pushed the fix/release-oneapi-image-runtime branch from b9cc797 to d645609 Compare September 30, 2026 08:23
The rebase onto #1626 kept master's open copy of
T-RELEASE-ONEAPI-IMAGE-B580-SIGSEGV-2026-09-29 next to this branch's
closing row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lusoris
lusoris merged commit 905a989 into master Sep 30, 2026
102 checks passed
@lusoris
lusoris deleted the fix/release-oneapi-image-runtime branch September 30, 2026 08:53
lusoris added a commit that referenced this pull request Sep 30, 2026
#1629 moved three sys.exit calls in dev/scripts/fetch-intel-neo.py, so
their HISS-07 fingerprints changed line. Recorded again from master's
baseline with praetor 25451d8: still 185 -> 431, the same 246 engine
findings; f41e74d still re-records 185 on this tree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lusoris added a commit that referenced this pull request Sep 30, 2026
Merge the RC2/RC3 disposition rows three-way by bug id: master (#1626)
had two RC3 rows from an earlier keep-both resolution, and this branch
adds three RC2 and six RC3 ids. Drop the open oneAPI B580 row #1629
closed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lusoris added a commit that referenced this pull request Sep 30, 2026
#1629 moved three sys.exit calls in dev/scripts/fetch-intel-neo.py, so
their HISS-07 fingerprints changed line. Recorded again from master's
baseline with praetor 25451d8: still 185 -> 431, the same 246 engine
findings; f41e74d still re-records 185 on this tree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lusoris added a commit that referenced this pull request Sep 30, 2026
)

* perf(sycl): read the shared frame in psnr, psnr_hvs and motion_v2

The SYCL psnr_hvs, psnr and motion_v2 twins now read the planes the SYCL
state uploads once per frame instead of converting and uploading their own
copies. Scores are bit-identical to the previous twins on an Arc B580 and a
UHD 770.

- Opt-in shared Cb/Cr planes in common.cpp (vmaf_sycl_shared_chroma_init /
  _upload, vmaf_sycl_get_shared_plane): the first chroma-reading twin of a
  frame packs the chroma into pinned staging and uploads it with one DMA per
  plane; later twins reuse it. Luma-only runs never allocate chroma.
- vmaf_sycl_queue_after_upload() gives twins on their own queue the input
  barriers the combined graph gets; a device-side slot fence orders each
  upload after the last readers of the slot it overwrites.
- psnr_hvs: no host float conversion or private upload; two work-items per
  8x8 block, one dispatch for all planes, per-block float expressions
  unchanged. 9- and 11-bit input now scores the raw sample like the CPU.
- motion_v2: runs the ADR-1371 pipeline on the shared luma and keeps the
  frame through its cur_copy; no host copy or private upload.
- psnr: chroma from the shared planes; one atomic per work-group.

At 3840x2160, psnr_hvs drops from 17.1 to 7.6 ms per frame on the B580 and
from 124 to 60 on the UHD 770, where psnr drops from 25.3 to 12.3.
ADR-1369, Research-1369.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: regenerate generated docs after rebasing onto master

Merge the RC2/RC3 disposition rows three-way by bug id: master (#1626)
had two RC3 rows from an earlier keep-both resolution, and this branch
adds three RC2 and six RC3 ids. Drop the open oneAPI B580 row #1629
closed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
lusoris added a commit that referenced this pull request Sep 30, 2026
The docs/state.md resolver let master's side of each conflict hunk win and
added only the bug ids master lacked. Mid-rebase "ours" is master plus the
branch commits already replayed, so on 2026-09-30 it kept master's Open copy
of bugs a branch had closed (#1627, #1629) and an earlier commit's text of a
row a later commit rewrote (#1625). Disposition rows conflicted on nearly
every rebase, and keep-both left master with two RC3 rows.

The resolver now reads the three index stages git keeps for the conflicted
path and merges them three-way (ADR-1383):

- rows and move tombstones are keyed by bug id, in the shapes
  check-state-md-rows.sh recognises; a row's state is its text plus its
  section, so a move, an edit, a close or a deletion on one side carries over
- disposition rows are keyed by their bold label; their id lists merge as
  sets and repeated labels are folded into one row
- every other line merges three-way by line; lines both sides added are
  kept, a line both sides added identically is kept once, and overlapping
  deletions are not a conflict
- a row both sides changed differently stops the tool with exit 1 and
  nothing written; --take NAME=ours|theirs settles it explicitly

It writes LF bytes, runs the row gate on its result and exits 3 when the
gate rejects it or an id sits in two disposition rows.

The test suite now builds throwaway repositories and drives real git rebase
conflicts through the tool. CI never ran the old test; it now runs in the
state.md row hygiene step of the Rules workflow and under the git fixture
isolation suite. Research-1383 replays the rebase conflicts of the PRs
merged since 2026-09-20 through both resolvers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lusoris added a commit that referenced this pull request Sep 30, 2026
#1629 moved three sys.exit calls in dev/scripts/fetch-intel-neo.py, so
their HISS-07 fingerprints changed line. Recorded again from master's
baseline with praetor 25451d8: still 185 -> 431, the same 246 engine
findings; f41e74d still re-records 185 on this tree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lusoris added a commit that referenced this pull request Sep 30, 2026
The docs/state.md resolver let master's side of each conflict hunk win and
added only the bug ids master lacked. Mid-rebase "ours" is master plus the
branch commits already replayed, so on 2026-09-30 it kept master's Open copy
of bugs a branch had closed (#1627, #1629) and an earlier commit's text of a
row a later commit rewrote (#1625). Disposition rows conflicted on nearly
every rebase, and keep-both left master with two RC3 rows.

The resolver now reads the three index stages git keeps for the conflicted
path and merges them three-way (ADR-1383):

- rows and move tombstones are keyed by bug id, in the shapes
  check-state-md-rows.sh recognises; a row's state is its text plus its
  section, so a move, an edit, a close or a deletion on one side carries over
- disposition rows are keyed by their bold label; their id lists merge as
  sets and repeated labels are folded into one row
- every other line merges three-way by line; lines both sides added are
  kept, a line both sides added identically is kept once, and overlapping
  deletions are not a conflict
- a row both sides changed differently stops the tool with exit 1 and
  nothing written; --take NAME=ours|theirs settles it explicitly

It writes LF bytes, runs the row gate on its result and exits 3 when the
gate rejects it or an id sits in two disposition rows.

The test suite now builds throwaway repositories and drives real git rebase
conflicts through the tool. CI never ran the old test; it now runs in the
state.md row hygiene step of the Rules workflow and under the git fixture
isolation suite. Research-1383 replays the rebase conflicts of the PRs
merged since 2026-09-20 through both resolvers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lusoris added a commit that referenced this pull request Sep 30, 2026
…1638)

* fix(dev): resolve docs/state.md rebase conflicts three-way by bug id

The docs/state.md resolver let master's side of each conflict hunk win and
added only the bug ids master lacked. Mid-rebase "ours" is master plus the
branch commits already replayed, so on 2026-09-30 it kept master's Open copy
of bugs a branch had closed (#1627, #1629) and an earlier commit's text of a
row a later commit rewrote (#1625). Disposition rows conflicted on nearly
every rebase, and keep-both left master with two RC3 rows.

The resolver now reads the three index stages git keeps for the conflicted
path and merges them three-way (ADR-1383):

- rows and move tombstones are keyed by bug id, in the shapes
  check-state-md-rows.sh recognises; a row's state is its text plus its
  section, so a move, an edit, a close or a deletion on one side carries over
- disposition rows are keyed by their bold label; their id lists merge as
  sets and repeated labels are folded into one row
- every other line merges three-way by line; lines both sides added are
  kept, a line both sides added identically is kept once, and overlapping
  deletions are not a conflict
- a row both sides changed differently stops the tool with exit 1 and
  nothing written; --take NAME=ours|theirs settles it explicitly

It writes LF bytes, runs the row gate on its result and exits 3 when the
gate rejects it or an id sits in two disposition rows.

The test suite now builds throwaway repositories and drives real git rebase
conflicts through the tool. CI never ran the old test; it now runs in the
state.md row hygiene step of the Rules workflow and under the git fixture
isolation suite. Research-1383 replays the rebase conflicts of the PRs
merged since 2026-09-20 through both resolvers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(research): record the final state.md replay numbers

The last replay ran after the overlapping-deletion fix; the older window now
has no refusals left from that defect and no conflict markers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(state): fold the duplicate RC3 disposition row and drop a stale _Updated line

Keep-both resolutions left two "RC3 performance and backend acceleration"
rows under "## First-release phase classification" and both versions of
union of their ids, exactly as the resolver folds repeated labels; every id
in the union is an open row. The first version of #1625's _Updated line is
dropped: the branch had already replaced it before its final rebase.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(ci): make check-state-md-rows.sh work with Debian 12's mawk

mawk 1.3.4 20200120, the awk of Debian 12, reads regex intervals such as
{0,2} and {4} literally. The gate's id pattern then matched no row, so it
reported "OK (0 id-bearing rows ...)" for every file and never ran its
duplicate-id or tombstone checks. The same mawk strips only one asterisk
with \*?\*?, so the optional bold is now spelled (\*\*|\*)? and the
verification date without intervals. CI's Ubuntu runner has a newer mawk and
was not affected; the self-test now passes on both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(state): say what the rebase left of the ledger cleanup

rebase this branch only drops the stale _Updated line; the _Updated entry
now says so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: regenerate generated docs after rebasing onto master

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: regenerate generated docs after rebasing onto master

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
lusoris added a commit that referenced this pull request Sep 30, 2026
#1629 moved three sys.exit calls in dev/scripts/fetch-intel-neo.py, so
their HISS-07 fingerprints changed line. Recorded again from master's
baseline with praetor 25451d8: still 185 -> 431, the same 246 engine
findings; f41e74d still re-records 185 on this tree.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type:bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants