Skip to content

fix(test): drop the stale HIP ADM should_fail markers and give the shared ADM tests a fixture that can fail - #1493

Merged
lusoris merged 1 commit into
masterfrom
fix/hip-adm-stale-should-fail
Sep 19, 2026
Merged

lusoris merged 1 commit into
masterfrom
fix/hip-adm-stale-should-fail

Conversation

@lusoris

@lusoris lusoris commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

Summary

Three HIP integer ADM tests on master hid behind stale should_fail : true markers. The markers cited the ADR-1154 picture-staging deferral, which ADR-1211 (PR #1370) ended two weeks ago, so on any machine with a ROCm device meson test reported test_hip_adm_parity as UNEXPECTEDPASS and exited non-zero. The other two, test_hip_adm_small_border and test_hip_adm_wide_rounding, "expectedly failed" in 40 ms for a reason the marker did not name: the shared CUDA/HIP sources ask for VMAF_integer_feature_adm3_score, which the HIP twin does not emit (no AIM pass, T-GPU-ADM-AIM-DEVICE-PASS-MISSING-SYCL-HIP-2026-09-05), so vmaf_feature_score_at_index() returned -EINVAL before any parity was compared.

Fixing the name alone was not enough. With adm3 skipped under HAVE_HIP, both tests passed bit-identical to the CPU, but planting the two pre-ADR-1167 defects back into adm_cm.hip showed the smooth-ramp fixture (row * 7 + col * 5) & 0xFF gave the contrast-masking kernel almost nothing to accumulate:

Planted defect in adm_cm.hip (gfx1036, ROCm 7.2.4) Ramp fixture: adm2 delta Textured fixture: adm2 delta
Border rows {1, 2, 3} + csf_a at row 2 when i == 0 && top <= 0 (ADR-1167 #1) 7.5e-06, passes 4.0e-04, test_hip_adm_small_border fails
Per-pixel shift_inner_accum (pre-ADR-1167 HIP shape, #2) 0.0 0.0
Per-pixel truncating shift, no rounding term 0.0 0.0
+1 unit per row after the row shift 0.0 0.0
Row shift dropped entirely (liveness probe) 1.3 / 1.8, fails fails
No defect 0.0 on every feature 0.0 on every feature, 20/20 runs per test

So the fixture is now a stateless lowbias32 texture at the same geometry (reference = full-range noise, distorted = reference + [-16, 15]; 160x96 keeps scale-3 top <= 0, 1920x144 keeps 60 warps per row). The tests name the failing feature and print every compared score.

The rounding-placement class the wide test was written for does not reach any emitted score: the CPU divides each accumulator by 2^(52 - shift_cub - shift_inner_accum) and casts to float before scoring, so per-row differences vanish. The wide test remains a valid gate for the 60-warp column striding and the block reduction (the liveness row); the placement itself needs an accumulator-level test, opened as T-ADM-CM-ROUNDING-PLACEMENT-UNOBSERVABLE-2026-09-19. Alternatives weighed for the fixture: tightening the tolerance (cannot help, the deltas are exactly zero), a new separate test (leaves the two shared tests unable to fail), or reading accum_global through a debug feature (the right tool for the rounding class, out of scope here).

Twins (ADR-1142): the CUDA arms of the two shared sources take the same fixture; CUDA and Metal twins emit adm3_score, the SYCL parity test already excludes it. The CUDA arms were syntax-checked (-fsyntax-only -DHAVE_CUDA=1) but not run; please run test_cuda_adm_small_border and test_cuda_adm_wide_rounding on the CUDA box before merge.

Overlap: PR #1476 (port/upstream-2026-09 stack) carries the identical adm3 guard hunk and the same marker removal; those hunks merge clean, the fixture and print changes are separate hunks. #1480 touches core/test/meson.build and core/src/feature/hip/AGENTS.md in other regions; the AGENTS section this PR rewrites is separated from #1480's hunk by unchanged lines.

Type

  • feat — new feature
  • fix — bug fix
  • perf — performance improvement
  • refactor — no behavior change
  • docs — documentation only
  • test — test-only
  • build / ci — tooling / infra
  • port — cherry-pick from upstream Netflix/vmaf
  • sycl / cuda / simd — backend-specific

Checklist

  • Commits follow Conventional Commits (the commit-msg hook enforces this).
  • make format && make lint is green locally: clang-format clean, clang-tidy 0 findings on both test files (-p build/hip), cppcheck 0 for the HIP and CUDA arms, assertion density PASS, praetorctl audit PASS, caveman check PASS on both AGENTS.md files.
  • Unit tests pass: meson test -C build/hip --num-processes 1 test_hip_adm_parity test_hip_adm_small_border test_hip_adm_wide_rounding → 3 OK, exit 0; each test 20/20 consecutive direct runs.
  • If I touched any SIMD/GPU code path, I ran /cross-backend-diff and the worst ULP is ≤ 2: no kernel change; HIP equals CPU exactly on every asserted feature (table above).
  • If I touched a feature extractor with SIMD/GPU twins, I either updated every twin or listed the gap under "Known follow-ups" below.
  • If I added a new .c / .cpp / .cu / .h / .hpp, it has the appropriate license header (see CONTRIBUTING.md): no new files.
  • If this is a breaking change, the commit message uses ! or BREAKING CHANGE: and the migration path is documented below: not breaking.
  • If this PR adds an ADR, the ADR row lives in docs/adr/_index_fragments/<NNNN-slug>.md and the slug is appended to docs/adr/_index_fragments/_order.txt: no ADR; a test fixture and marker fix.

Bug-status hygiene (ADR-0165)

Netflix golden-data gate (ADR-0024)

  • I did not modify any assertAlmostEqual(...) score in the Netflix golden Python tests.
  • If I believe a golden value must change, I have explained why below AND pinged @lusoris for a CODEOWNERS exception: no CPU score changes.

Cross-backend numerical results

textured fixture, gfx1036 vs scalar CPU (delta in score units)
test_hip_adm_small_border  adm2 0.95237863 = 0.95237863 (0.0)  scale3 0.94376788 = 0.94376788 (0.0)
test_hip_adm_wide_rounding adm2 0.95597338 (0.0)  scale0 0.95670021 (0.0)  scale1 0.95396689 (0.0)  scale2 0.95541070 (0.0)  scale3 0.95879157 (0.0)

Deep-dive deliverables (ADR-0108)

  • Research digest — no digest needed: trivial marker and test-fixture fix; the measurements are in this description and in the state rows.
  • Decision matrix — no alternatives: only-one-way fix (a stale marker is removed, a name the twin never emits is skipped; the fixture alternatives are weighed in the summary and need no ADR).
  • AGENTS.md invariant note — core/test/AGENTS.md and core/src/feature/hip/AGENTS.md.
  • Reproducer / smoke-test command — below.
  • CHANGELOG fragment — changelog.d/fixed/hip-adm-stale-should-fail.md.
  • Rebase note — docs/rebase-notes.md.

Reproducer

meson setup build/hip core -Denable_hip=true -Denable_hipcc=true -Denable_cuda=false \
  -Denable_sycl=false -Db_lto=false --buildtype=release
ninja -C build/hip
meson test -C build/hip --num-processes 1 test_hip_adm_parity test_hip_adm_small_border test_hip_adm_wide_rounding
# before this change: 1 UNEXPECTEDPASS + 2 EXPECTEDFAIL ("vmaf_feature_score_at_index failed"), exit 1
# after: 3 OK, exit 0

Known follow-ups

  • T-ADM-CM-ROUNDING-PLACEMENT-UNOBSERVABLE-2026-09-19: an accumulator-level test for where the CM kernels apply shift_inner_accum; no score-level gate can see it (CUDA, SYCL and Metal twins included).
  • T-GPU-ADM-AIM-DEVICE-PASS-MISSING-SYCL-HIP-2026-09-05: the HIP and SYCL integer ADM twins still have no AIM pass, so adm3_score stays with the CPU twin.
  • integer_adm_hip keeps .flags = 0: model-driven dispatch under --backend hip still runs the CPU adm; the twin runs when named (--feature adm_hip).
  • The CUDA arms of test_adm_small_border.c / test_adm_wide_rounding.c need a run on the RTX 4090 (not driven from this session).

@lusoris
lusoris force-pushed the fix/hip-adm-stale-should-fail branch from 8180029 to cdfd513 Compare September 19, 2026 12:20
@github-actions github-actions Bot added the type:bug Something isn't working label Sep 19, 2026
@lusoris
lusoris force-pushed the fix/hip-adm-stale-should-fail branch from cdfd513 to 58fb4b2 Compare September 19, 2026 14:32
@lusoris

lusoris commented Sep 19, 2026

Copy link
Copy Markdown
Contributor Author

The Docs failure here was never this branch's doing, and it is being fixed in #1494.

The docs-lint job has a 10-minute ceiling while mkdocs build --strict over this doc tree measured 9m31s, 9m21s and 5m21s on its last three green runs. On the ubuntu-26.04 image it crosses 10 minutes, so the job is cancelled mid-build and the required-checks aggregator reads the cancellation as a failure. On this branch's run the generated-docs freshness step passed and the build step was cancelled at 12:55:07Z after exactly the ceiling, while all eleven sibling jobs in the same run finished successfully, several more than twenty minutes later. It reproduces on master's own first run after the runner bump merged.

#1494 raises that ceiling to 20 minutes. This branch is rebased onto current master; once #1494 lands, this should go green on its own.

@lusoris
lusoris force-pushed the fix/hip-adm-stale-should-fail branch from 58fb4b2 to be584d6 Compare September 19, 2026 15:22
… ADM tests a fixture that can fail

test_hip_adm_parity, test_hip_adm_small_border and test_hip_adm_wide_rounding
kept `should_fail : true` from the ADR-1154 staging deferral after ADR-1211
(PR #1370) fixed it, so `meson test` on a HIP device exited non-zero on the
unexpected pass. The other two failed in 40 ms because the shared CUDA/HIP
sources asked for VMAF_integer_feature_adm3_score, which the HIP twin does
not emit (-EINVAL), so no parity was ever compared.

Skip adm3 under HAVE_HIP, name the failing feature and print every compared
score. Replace the smooth-ramp fixture with a stateless lowbias32 texture at
the same geometry: with the pre-ADR-1167 border defect planted back into
adm_cm.hip the ramp moved adm2 by 7.5e-6 against the 1e-4 gate, the texture
moves it by 4.0e-4 and fails the test. Clean HIP stays bit-identical to the
CPU on every feature (20/20 runs per test, gfx1036, ROCm 7.2.4).

The rounding placement the wide test was written for is not observable in
any emitted score (the CPU casts the scaled accumulator to float); opened as
T-ADM-CM-ROUNDING-PLACEMENT-UNOBSERVABLE-2026-09-19. Closes
T-GAP-HIP-INTEGER-ADM-PICTURE-STAGING-DEFERRED-2026-09-02 and
T-HIP-ADM-TESTS-STALE-SHOULD-FAIL-2026-09-18.
@lusoris
lusoris force-pushed the fix/hip-adm-stale-should-fail branch from be584d6 to a78630f Compare September 19, 2026 19:16
@lusoris
lusoris merged commit e3570ef into master Sep 19, 2026
82 checks passed
@lusoris
lusoris deleted the fix/hip-adm-stale-should-fail branch September 19, 2026 20:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type:bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant