test(metal): make the macOS tester report measure every open Metal row (ADR-1496) by lusoris · Pull Request #1918 · VMAFx/vmafx

lusoris · 2026-10-03T11:41:51Z

Summary

The next run of the macOS tester bundle on the outside tester's Apple M4 has to measure every open Metal row of docs/state.md in one go; until now it compared default-option scores on four fixtures and ran Metal unit tests that compared at places=4 and stopped at the first failure. This PR makes that run measure each row and say so (ADR-1496):

Parity gate: a metal backend in both gate scripts and --hold-exact <backend> (cells of that backend at tolerance 0 and --precision max, ciede at the LIBM_TWINS bound, before any exact_twins.d fragment lists it). Metal leaves UNGATED_BACKENDS (T-GATE-NO-METAL-BACKEND-2026-10-02).
Metal parity tests: all 18 test_metal_*_parity compare with == on the CUDA/HIP/SYCL twins' cases (the shared *_twin_parity.h headers) plus the cases the rows need: full-range 16-bit content, 10/12/16-bit input with large differences, option sets, identical pairs, one frame, frames below 16 and 17 px, adm_noise_weight=0 on a flat 16-bit frame, adm_enhn_gain_limit 1.2 and 1.5, motion3 and the motion SAD score. Every case runs after a failure and prints an @case verdict line (core/test/metal_twin.h). Without a Metal device every case skips (exit 77), so the hosted macOS job stays green on today's twins. test_metal_twin_option_parity compares every Metal twin's option table, provided features and TEMPORAL flag with the CPU's.
Self-tests: the same sources build on every host with the CPU in the twin's place (test_metal_selftest_*, suites fast and metal-selftest): a wrong fixture, key or option string fails CI, not the tester's run.
Kernels: every .metal compiles with -std=metal3.1 -mmacosx-version-min=14.0; without a target the offline compiler stamps the runner SDK's macOS and the metallib refuses to load on an older macOS.
Report (schema 2): runs the staged gate on every fixture (metal_gate, part of the verdict), keeps per-case verdicts (unit_tests.cases), and evaluates tools/rc1-tester/image/metal-rows.json (metal_rows: pass, fail or not measured per row).
Release assets: the bundle script no longer leaves the downloaded interpreter archive where the workflow publishes every *.tar.gz (tester-20261003-c12763f3 released and signed pbs.tar.gz; T-TESTER-BUNDLE-PUBLISHES-INTERPRETER-ARCHIVE-2026-10-03, found and fixed here).

No Metal twin changes here; the ports are the follow-up PR. A bundle built from this commit reports today's Metal defects as failing cases and rows.

Type

test — test-only
build / ci — tooling / infra

Checklist

Commits follow Conventional Commits (the commit-msg hook enforces this).
make format && make lint is green locally (pre-commit: black, ruff, markdownlint, semgrep, citations).
Unit tests pass: the 18 Metal self-tests (meson test --suite metal-selftest: 18 OK), the Python suites below.
If I touched any SIMD/GPU code path, I ran /cross-backend-diff and the worst ULP is ≤ 2. — no GPU code path changed; no Apple device on this host.
If I touched a feature extractor with SIMD/GPU twins, I either updated every twin or listed the gap under "Known follow-ups" below.
If I added a new .c / .cpp / .cu / .h / .hpp, it has the appropriate license header (see CONTRIBUTING.md).
If this is a breaking change, the commit message uses ! or BREAKING CHANGE: and the migration path is documented below. — not breaking: report schema 1 stays valid.
If this PR adds an ADR, the ADR row lives in docs/adr/_index_fragments/<NNNN-slug>.md and the slug is appended to docs/adr/_index_fragments/_order.txt.

Bug-status hygiene (ADR-0165)

docs/state.md updated: the leads of the 18 open Metal rows say which bundle cases measure them; T-GATE-NO-METAL-BACKEND-2026-10-02 records the gate backend. No Metal row closed: they close with the tester's report. T-TESTER-BUNDLE-PUBLISHES-INTERPRETER-ARCHIVE-2026-10-03 found and closed under Recently closed.

Netflix golden-data gate (ADR-0024)

I did not modify any assertAlmostEqual(...) score in the Netflix golden Python tests.

Deep-dive deliverables (ADR-0108)

Research digest — no digest needed: the decision and its alternatives are in ADR-1496; the one external fact (Metal deployment-target stamping, MSL 4.1 §1.6.10) is cited there.
Decision matrix — ADR-1496 ## Alternatives considered (report-as-gate, default tolerances, early fragments, keep old tests, per-case processes, compiler-default target).
AGENTS.md invariant note — core/test/AGENTS.d/metal-parity-tests.md (new), scripts/ci/AGENTS.d/parity-gate.md, tools/rc1-tester/AGENTS.md item 8.
Reproducer / smoke-test command — below.
CHANGELOG fragment — changelog.d/added/metal-report-full-measurement.md.
Rebase note — docs/rebase-notes.md, "The macOS tester bundle measures every open Metal row".

Reproducer

meson setup build core -Denable_cuda=false -Denable_sycl=false && ninja -C build
python3 scripts/ci/run_meson_test.py -- -C build --suite metal-selftest
python3 -m pytest -q tools/rc1-tester/tests scripts/ci/test_cross_backend_parity_gate.py \
  core/test/test_metal_report_rows_contract.py core/test/test_metal_shader_build_contract.py \
  core/test/test_parity_gate_covers_registered_twins.py
# On an Apple-silicon Mac with a Metal build:
python3 scripts/ci/cross_backend_parity_gate.py --vmaf-binary build/tools/vmaf \
  --reference python/test/resource/yuv/src01_hrc00_576x324.yuv \
  --distorted python/test/resource/yuv/src01_hrc01_576x324.yuv \
  --width 576 --height 324 --backends cpu metal --hold-exact metal

Local results: 18/18 Metal self-tests OK (1.5 s wall); 297 Python tests passed; bash 3.2 bundle-build test passed; MSVC-ism preflight passed.

Known follow-ups

The Metal ports themselves (fix/metal-twins-exact), then a bundle from master and the tester's second run (see ADR-1496 consequences).
psnr_hvs_metal, integer_cambi_metal, ssimulacra2_metal and the integer twins outside the rows are now held to == too; no row records a defect there, the run will tell.

+    unsigned bad = 0u;
+    for (unsigned i = 0; i < sc->frames; i++) {
+        for (unsigned k = 0; k < key_count(sc); k++) {
+            if (isfinite(cpu[i][k]) && cpu[i][k] == twin[i][k]) {


+        for (unsigned k = 0; k < N_KEYS; k++) {
+            char name[NAME_BYTES];
+            key_name(k, name);
+            if (!key_active(c, k) || (isfinite(cpu[i][k]) && cpu[i][k] == gpu[i][k])) {


    }
-    if (!result) {
-        result = collect_float_ms_ssim_scores(vmaf, false, out_y, out_cb, out_cr);
+    if (c->expect_cpu != 0.0 && cpu[0][0] != c->expect_cpu) {


+                      c->expect_cpu);
    }
+    mu_assert("the CPU's float_ms_ssim is no longer the value the fixture was recorded with",
+              c->expect_cpu == 0.0 || cpu[0][0] == c->expect_cpu);


+    unsigned differing = 0u;
+    for (unsigned i = 0; i < c->frames; i++) {
+        for (unsigned k = 0; k < (c->lcs ? N_KEYS : 1u); k++) {
+            if (isfinite(cpu[i][k]) && cpu[i][k] == gpu[i][k]) {


+    const bool on_cpu = g.w < expected_min_dim() || g.h < expected_min_dim();
+    for (unsigned i = 0; i < NUM_FRAMES; i++) {
+        for (unsigned k = 0; k < NUM_SCALES; k++) {
+            if (cpu[i][k] == gpu[i][k]) {


+    unsigned bad = 0u;
+    for (unsigned i = 0; i < sc->frames; i++) {
+        for (unsigned k = 0; k < key_count(sc); k++) {
+            if (isfinite(cpu[i][k]) && cpu[i][k] == twin[i][k]) {


-        char *feed_err = feed_fixture_pair(vmaf, i);
-        if (feed_err)
-            return feed_err;
+        if (isfinite(cpu[i]) && cpu[i] == gpu[i]) {


+    double cpu[NUM_FRAMES] = {0.0};
+    mu_assert("ssimulacra2_metal differs from the CPU extractor", exact_mismatches(&c, cpu) == 0u);
+    mu_assert("ssimulacra2 fixture frames must score differently",
+              mu_skipped || (cpu[0] != cpu[1] && cpu[1] != cpu[2]));


+    double cpu[NUM_FRAMES] = {0.0};
+    mu_assert("ssimulacra2_metal differs from the CPU extractor", exact_mismatches(&c, cpu) == 0u);
+    mu_assert("ssimulacra2 fixture frames must score differently",
+              mu_skipped || (cpu[0] != cpu[1] && cpu[1] != cpu[2]));


…w (ADR-1496) (#1918) * test(metal): make the macOS tester report measure every open Metal row (ADR-1496) A tester run of the macOS bundle is the only place a Metal twin meets an Apple device, and the next run has to measure every open Metal row of docs/state.md at once. - Parity gate: a `metal` backend in both gate scripts and `--hold-exact`, which compares a backend's cells exactly at `--precision max` (ciede at the LIBM_TWINS bound) before a fragment lists it. Metal leaves UNGATED_BACKENDS (T-GATE-NO-METAL-BACKEND-2026-10-02). - Metal parity tests: every test_metal_*_parity compares with `==` on the CUDA, HIP and SYCL twins' cases (shared *_twin_parity.h headers) plus the cases the open rows need, runs every case after a failure and prints one `@case` verdict line per case (core/test/metal_twin.h). Without a device every case skips. test_metal_twin_option_parity compares every Metal twin's option table, provided features and TEMPORAL flag with the CPU's. The same sources build on every host as self-tests with the CPU in the twin's place (suite metal-selftest, also fast). - Kernels: every .metal compiles with -std=metal3.1 -mmacosx-version-min=14.0, so the metallib loads on the bundle's macOS 14 floor instead of carrying the runner SDK's deployment target. - Tester report (schema 2): runs the staged gate on every fixture (`metal_gate`, a check of the verdict), keeps per-case verdicts (`unit_tests.cases`) and evaluates tools/rc1-tester/image/metal-rows.json (`metal_rows`: per row pass, fail or not measured). The bundle lists every Metal parity test and carries the gate, its fragments and the ADRs they cite. - Contract tests: the row map against docs/state.md, the tests, the unit list and the gate; the kernels' target arguments against the bundle. The 18 Metal rows say which cases measure them; none is closed: they close with the tester's report. * test(metal): write the psnr aggregate files with mkstemp, not under a getenv() path test_metal_integer_psnr_parity reads the apsnr_* aggregates back from the JSON output. It built the file name from getenv("TMPDIR"), which the cpu tidy lane reports (concurrency-mt-unsafe, 3 findings in a new file). It now takes mkstemp() in /tmp and GetTempPathA() on Windows, the pattern of core/test/AGENTS.d/temp-files-and-output.md. Tidy lane cpu on the file: 0. * fix(ci): keep the interpreter archive out of the macOS tester bundle's release assets build-macos-tester-bundle.sh downloaded the python-build-standalone archive into its output directory, and macos-tester-bundle.yml uploads, attests, signs and releases every *.tar.gz there: tester-20261003-c12763f3 carries pbs.tar.gz and its cosign bundle, a third-party file signed with the project's identity. The script removes the archive and its extraction directory once the interpreter is staged. test_bash32_compat fails without the removal (T-TESTER-BUNDLE-PUBLISHES-INTERPRETER-ARCHIVE-2026-10-03). * test(metal): update contract tests for unified metal parity harness * test(metal): run the float_ms_ssim_chroma gate cell on Metal and count the gate's own skips as neutral #1917 added the parity gate's float_ms_ssim_chroma feature, which the gate skips (status SKIP) where a chroma plane is below 176 pixels, as on the 576x324 fixtures. The macOS bundle runs it on Metal (metal-rows.json, the float_ms_ssim row), test_metal_report_rows_contract holds the gate list to the features with a Metal twin, and a SKIP cell neither fails the report's metal_gate nor counts as evidence for a row. * test(metal): adapt float_moment parity test to master exact sum API

… fit the sanitizer job #1918 put the Metal parity tests on every host as self-tests, and three hosted jobs failed on them. MSVC: test_metal_integer_motion_parity.c and test_metal_motion_v2_parity.c named a scenario SC_DEFAULT, which winuser.h defines as 0xF160; the pthread shim includes windows.h, so the definition read `static const Scenario 0xF160 = {` (C2059). The scenarios are SC_DEFAULTS. A MinGW `-fsyntax-only -include windows.h` check reproduces the error on the old files and is clean on every test_metal_*_parity.c now. macOS: test_metal_selftest_integer_psnr died with SIGSEGV in expect_aggregate() on the hosted runner (release build with LTO), in the path that wrote the context's JSON output to a mkstemp() file and read apsnr_* back. The test passes on Linux under ASan and UBSan, so the fault inside that path was not isolated here. The test now reads the aggregate from the context's feature collector (vmaf_feature_collector_get() of libvmaf_priv.h and vmaf_feature_collector_get_aggregate()), the double the writer prints, with no file, temporary directory or windows.h. Sanitizers: the job failed on test_float_moment_sum (ADR-1497) reaching its 120 s timeout under the debug ASan + UBSan build on the hosted runner, twice. It takes 35 s on a workstation in that configuration; the timeout is 600 s. The job's own invocation, run here, enumerates 293 tests and passes every Metal self-test.

… fit the sanitizer job (#1925) * fix(test): make the Metal self-tests build on MSVC, pass on macOS and fit the sanitizer job #1918 put the Metal parity tests on every host as self-tests, and three hosted jobs failed on them. MSVC: test_metal_integer_motion_parity.c and test_metal_motion_v2_parity.c named a scenario SC_DEFAULT, which winuser.h defines as 0xF160; the pthread shim includes windows.h, so the definition read `static const Scenario 0xF160 = {` (C2059). The scenarios are SC_DEFAULTS. A MinGW `-fsyntax-only -include windows.h` check reproduces the error on the old files and is clean on every test_metal_*_parity.c now. macOS: test_metal_selftest_integer_psnr died with SIGSEGV in expect_aggregate() on the hosted runner (release build with LTO), in the path that wrote the context's JSON output to a mkstemp() file and read apsnr_* back. The test passes on Linux under ASan and UBSan, so the fault inside that path was not isolated here. The test now reads the aggregate from the context's feature collector (vmaf_feature_collector_get() of libvmaf_priv.h and vmaf_feature_collector_get_aggregate()), the double the writer prints, with no file, temporary directory or windows.h. Sanitizers: the job failed on test_float_moment_sum (ADR-1497) reaching its 120 s timeout under the debug ASan + UBSan build on the hosted runner, twice. It takes 35 s on a workstation in that configuration; the timeout is 600 s. The job's own invocation, run here, enumerates 293 tests and passes every Metal self-test.

… fit the sanitizer job #1918 put the Metal parity tests on every host as self-tests, and three hosted jobs failed on them. MSVC: test_metal_integer_motion_parity.c and test_metal_motion_v2_parity.c named a scenario SC_DEFAULT, which winuser.h defines as 0xF160; the pthread shim includes windows.h, so the definition read `static const Scenario 0xF160 = {` (C2059). The scenarios are SC_DEFAULTS. A MinGW `-fsyntax-only -include windows.h` check reproduces the error on the old files and is clean on every test_metal_*_parity.c now. macOS: test_metal_selftest_integer_psnr died with SIGSEGV in expect_aggregate() on the hosted runner (release build with LTO), in the path that wrote the context's JSON output to a mkstemp() file and read apsnr_* back. The test passes on Linux under ASan and UBSan, so the fault inside that path was not isolated here. The test now reads the aggregate from the context's feature collector (vmaf_feature_collector_get() of libvmaf_priv.h and vmaf_feature_collector_get_aggregate()), the double the writer prints, with no file, temporary directory or windows.h. Sanitizers: the job failed on test_float_moment_sum (ADR-1497) reaching its 120 s timeout under the debug ASan + UBSan build on the hosted runner, twice. It takes 35 s on a workstation in that configuration; the timeout is 600 s. The job's own invocation, run here, enumerates 293 tests and passes every Metal self-test.

github-actions Bot added the type:test Test-only change label Oct 3, 2026

lusoris force-pushed the test/metal-report-full-measurement branch 2 times, most recently from b98f8d0 to 4f79611 Compare October 3, 2026 12:54

lusoris mentioned this pull request Oct 3, 2026

fix(metal): port the Metal twins to the exact designs of their CUDA, HIP and SYCL twins (ADR-1498) #1921

Merged

17 of 18 tasks

github-advanced-security Bot found potential problems Oct 3, 2026

View reviewed changes

lusoris force-pushed the test/metal-report-full-measurement branch from 4f79611 to 5e21f47 Compare October 3, 2026 13:14

lusoris force-pushed the test/metal-report-full-measurement branch from 5e21f47 to a8723a1 Compare October 3, 2026 13:20

lusoris merged commit a8723a1 into master Oct 3, 2026
51 of 54 checks passed

lusoris deleted the test/metal-report-full-measurement branch October 3, 2026 13:21

lusoris deployed to github-pages October 3, 2026 13:56 — with GitHub Actions View deployment

This was referenced Oct 3, 2026

fix(simd): add the NEON and SVE2 float_moment lanes in the scalar's order so aarch64 returns the scalar bits past 2^53 units #1923

Merged

fix(test): make the Metal self-tests build on MSVC, pass on macOS and fit the sanitizer job #1925

Merged

Navigation Menu

Repository navigation

test(metal): make the macOS tester report measure every open Metal row (ADR-1496) - #1918

test(metal): make the macOS tester report measure every open Metal row (ADR-1496)#1918
lusoris merged 1 commit into
masterfrom
test/metal-report-full-measurement

lusoris commented Oct 3, 2026 •

edited

Loading

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

lusoris commented Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Type

Checklist

Bug-status hygiene (ADR-0165)

Netflix golden-data gate (ADR-0024)

Deep-dive deliverables (ADR-0108)

Reproducer

Known follow-ups

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

lusoris commented Oct 3, 2026 •

edited

Loading