Skip to content

test(metal): add CPU-vs-Metal parity tests for remaining 4 kernels (round 3) - #447

Merged
lusoris merged 1 commit into
masterfrom
test/metal-kernel-coverage-round3
May 31, 2026
Merged

lusoris merged 1 commit into
masterfrom
test/metal-kernel-coverage-round3

Conversation

@lusoris

@lusoris lusoris commented May 31, 2026

Copy link
Copy Markdown
Contributor

Summary

Round 3 of Metal kernel test coverage. With this PR all 8 registered Metal extractors have a real per-kernel CPU-vs-Metal score parity gate:

New tests (all core/test/, all suite : ['fast', 'gpu']):

Kernel CPU twin Score key(s) Tolerance ADR
integer_motion_metal motion VMAF_integer_feature_motion2_score 1e-4 ADR-0214
float_motion_metal float_motion VMAF_feature_motion2_score 1e-4 ADR-0214 / PR #1018
float_moment_metal float_moment 4 keys (ref1st, dis1st, ref2nd, dis2nd) 1e-4 ADR-0214
float_ms_ssim_metal float_ms_ssim float_ms_ssim 1e-3 ADR-0589 (SSIM family)

Skip path. Each test calls vmaf_metal_state_init. On Linux / Windows / Intel Mac that returns -ENODEV; the test emits [skip: no Metal device] and passes cleanly. Apple-Family-7+ macOS CI lanes run the live kernels and assert the per-kernel tolerance. Pattern mirrors PR #379 and core/test/test_sycl_motion3_parity.c.

Wiring. 4 new executable() + test() entries appended inside the existing enable_metal guard in core/test/meson.build, immediately after the round-2 block (or after test_metal_install_header pre-PR-#379-merge) — no boundary edits, so upstream syncs cannot conflict here.

PR overlap (avoided)

Per-kernel coverage count (post-merge)

Round PR Kernels covered
1 (registration) #351 8 (discoverability only)
2 (parity, batch A) #379 4 (motion_v2, integer_psnr, float_psnr, float_ssim)
3 (parity, batch B) this PR 4 (integer_motion, float_motion, float_moment x4 keys, float_ms_ssim)
Total real-parity 8 / 8

Test plan

  • CPU-only build green: cd core && meson setup build-cpu -Denable_cuda=false -Denable_sycl=false -Denable_metal=disabled succeeds (new tests gated off on non-Metal).
  • pre-commit run --files on every touched file -> all hooks pass (clang-format, copyright header, semgrep, conventional-commit, etc.).
  • bash scripts/ci/check-copyright.sh core/test/test_metal_*_parity.c -> silent (clean).
  • bash scripts/ci/assertion-density.sh core/test/test_metal_*_parity.c -> PASS (every fork-added function ≥20 lines has ≥1 assert).
  • clang-format --dry-run --Werror core/test/test_metal_*_parity.c -> no diff.
  • Apple Silicon CI lane (macOS Metal): meson test -C build --suite=fast test_metal_integer_motion_parity test_metal_float_motion_parity test_metal_float_moment_parity test_metal_float_ms_ssim_parity exercises the live kernels and asserts parity. (Will run automatically when the macOS Metal CI lane picks up this PR — Apple-Family-7+ device required.)

ADR-0108 deliverables

  • Research digest: no digest needed: trivial test-only addition (follow-up to PR test(gpu): GPU backend kernel parity coverage gap-fill (CUDA/HIP/SYCL/Metal) #351 / PR test(metal): add CPU-vs-Metal parity tests for 4 kernels #379 documented gap; same pattern, different kernels).
  • Decision matrix: no alternatives: only-one-way fix (the synthetic-fixture + skip-on-ENODEV pattern is the established test_sycl_motion3_parity.c + PR test(metal): add CPU-vs-Metal parity tests for 4 kernels #379 precedent; tolerance choices inherit from existing ADR-0214 / ADR-0589).
  • AGENTS.md invariant note: no rebase-sensitive invariants in this PR. PR test(metal): add CPU-vs-Metal parity tests for 4 kernels #379 already extends core/test/AGENTS.md "GPU tests must skip gracefully" rule to mention vmaf_metal_state_init and Intel-Mac -ENODEV; this PR consumes that contract without re-stating it.
  • Reproducer / smoke-test command:
    cd core && meson setup build-cpu -Denable_cuda=false -Denable_sycl=false -Denable_metal=disabled
    ninja -C build-cpu
    # On Apple Silicon:
    cd core && meson setup build -Denable_metal=enabled
    meson test -C build --suite=fast test_metal_integer_motion_parity test_metal_float_motion_parity test_metal_float_moment_parity test_metal_float_ms_ssim_parity
  • CHANGELOG fragment: changelog.d/added/metal-kernel-coverage-round3.md.
  • Rebase note: docs/rebase-notes.md -> "Metal kernel parity tests round 3 (2026-05-31)" entry added; rebase impact NONE (all new file paths are fork-local Metal-only, the meson edit lives inside the fork-added enable_metal block).

docs/state.md updated with the T-METAL-KERNEL-PARITY-ROUND3-2026-05-31 row under "Recently closed" (per CLAUDE §12 r13).

Cross-references

🤖 Generated with Claude Code

Round 3 of Metal kernel test coverage. PR #351 closed the registration
audit (all 8 extractors discoverable) and PR #379 added parity tests
for the 4 highest-priority kernels (motion_v2, integer_psnr, float_psnr,
float_ssim). This commit fills the remaining 4 gaps so every registered
Metal extractor now has a real per-kernel CPU-vs-Metal score gate.

New tests (all in core/test/, all skip on -ENODEV):
- test_metal_integer_motion_parity.c   -> motion2_score (1e-4, ADR-0214)
- test_metal_float_motion_parity.c     -> motion2_score (1e-4, ADR-0214)
- test_metal_float_moment_parity.c     -> 4 moment keys (1e-4, ADR-0214)
- test_metal_float_ms_ssim_parity.c    -> ms_ssim       (1e-3, ADR-0589)

Pattern mirrors PR #379 / test_sycl_motion3_parity.c: synthetic 256x144
YUV420P fixture fed through both the CPU twin and the Metal extractor;
assert places=4 (1e-4) parity per ADR-0214, except float_ms_ssim which
inherits the 1e-3 SSIM-family bound from ADR-0589.

Skip path: when vmaf_metal_state_init returns -ENODEV (Linux, Windows,
Intel Mac), each test emits "[skip: no Metal device]" and passes
cleanly. Runs the live kernel only on Apple-Family-7+ macOS CI lanes.

Wiring: 4 new executable() + test() entries appended inside the
existing enable_metal guard in core/test/meson.build (after the
round-2 block, or after test_metal_install_header pre-merge), suite
['fast', 'gpu'] for consistency with the existing Metal smoke / round-2
parity tests.

ADR-0108 deliverables:
- Research digest: no digest needed: trivial test-only addition
- Decision matrix: no alternatives: only-one-way fix (synthetic-fixture
  + skip-on-ENODEV is the established test_sycl_motion3_parity.c +
  PR #379 precedent)
- AGENTS.md invariant note: no rebase-sensitive invariants in this PR;
  PR #379 already updates core/test/AGENTS.md for the Metal -ENODEV
  skip rule
- Reproducer: see PR description
- CHANGELOG fragment: changelog.d/added/metal-kernel-coverage-round3.md
- Rebase note: docs/rebase-notes.md "Metal kernel parity tests round 3
  (2026-05-31)"

State.md updated with T-METAL-KERNEL-PARITY-ROUND3-2026-05-31 row
under Recently closed (CLAUDE r13).

Refs: PR #294, PR #308, PR #351 (registration audit), PR #379 (round 2)
Cross-refs: ADR-0214, ADR-0361, ADR-0421, ADR-0589

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@lusoris

lusoris commented May 31, 2026

Copy link
Copy Markdown
Contributor Author

parity sweep audit: clean, no symmetric bug.

Verified all 4 new test files use the correct Metal-suffixed registry names:

  • test_metal_integer_motion_parity.c -> integer_motion_metal (registered at core/src/feature/metal/integer_motion_metal.mm:324)
  • test_metal_float_motion_parity.c -> float_motion_metal (core/src/feature/metal/float_motion_metal.mm:335)
  • test_metal_float_moment_parity.c -> float_moment_metal (core/src/feature/metal/float_moment_metal.mm:266)
  • test_metal_float_ms_ssim_parity.c -> float_ms_ssim_metal (core/src/feature/metal/float_ms_ssim_metal.mm:553)

CPU baselines (motion, float_motion, float_moment, float_ms_ssim) all resolve correctly in the CPU registry. The integer_motion_metal test correctly pairs against CPU 'motion' (integer path in core/src/feature/integer_motion.c). No fix needed.

@lusoris
lusoris marked this pull request as ready for review May 31, 2026 04:22
@lusoris
lusoris enabled auto-merge (squash) May 31, 2026 04:22
@lusoris
lusoris merged commit f01b7d2 into master May 31, 2026
94 of 101 checks passed
@lusoris
lusoris deleted the test/metal-kernel-coverage-round3 branch May 31, 2026 04:45
@lusoris lusoris added this to the 1.0.0 — First release milestone Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant