cuda: use the CPU's reflect-101 edge mirror in the motion kernels

[lusoris]

The CUDA motion kernels mirror an out-of-range index with sup - (out - sup + 1), which is 2 * sup - idx - 1. The CPU path has used reflect-101, 2 * size - idx - 2, since a44e5e6, and the CUDA copy was not updated. integer_motion2 and VMAF from the CUDA extractor therefore differ from the CPU wherever the edge samples carry detail (the two checkerboard test pairs are unaffected). This addresses the first root cause of #1562. The second one, the rounding order of blurring each frame before subtracting (a residual of about 1e-5 per frame), is not part of this change.

Cause

mirror() in libvmaf/src/feature/cuda/integer_motion/motion_score.cu is called by both calculate_motion_score_kernel_8bpc and calculate_motion_score_kernel_16bpc for every filter tap that leaves the picture. For idx >= sup it returns sup - (out - sup + 1). a44e5e6 changed the same expression in the CPU path (libvmaf/src/feature/integer_motion.c) from height - (i_tap - height + 1) to height - (i_tap - height + 2), which is the mirror() there today. The two outermost rows and columns of every frame are therefore filtered with the wrong neighbours on the GPU. The effect is a border effect and shrinks with the frame width.

Reproducer

Master 8e7a1ac4e, release build with -Denable_cuda=true, RTX 4090, CUDA 13.4. Netflix src01 pair (48 frames, 8-bit) and its yuv420p10le version (3 frames), vmaf_v0.6.1:

vmaf -r src01_hrc00_576x324.yuv -d src01_hrc01_576x324.yuv -w 576 -h 324 -p 420 -b 8 \
    --model version=vmaf_v0.6.1 -q --json --gpumask 4294967295 -o cpu.json   # CPU
vmaf ... --gpumask 0 -o gpu.json                                              # CUDA

Largest absolute per-frame difference between the two, and the pooled VMAF mean (CPU / CUDA):

input	integer_motion2 master	this branch	vmaf master	this branch	pooled VMAF master	this branch
8-bit, 48 frames	2.62e-3	1.3e-5	2.85e-3	1.4e-5	76.667831 / 76.668905	76.667831 / 76.66783
10-bit, 3 frames	1.35e-3	3e-6	1.51e-3	3e-6	82.564227 / 82.56523	82.564227 / 82.564225

The CLI prints six digits. What remains on this branch is the blur-then-subtract rounding of the second root cause.

Fix

The offset in mirror() changes from + 1 to + 2, as in integer_motion.c.

Tests

test_cuda_motion (new, libvmaf/test/, needs a CUDA device) runs motion on the CPU and motion_cuda on four 96x64 frames of noise at 8 and 10 bits and requires VMAF_integer_feature_motion2_score to agree within 1e-3 on every frame. On unpatched master the 8-bit case fails with a difference of 5.0e-2; on this branch the differences are 1.7e-5 (8-bit) and 1.9e-5 (10-bit).

Validation

x86-64 Linux, GCC 16.2.1, nvcc 13.4, RTX 4090, release build on master 8e7a1ac4e with -Denable_cuda=true.

Rebased on master 9e48141b (2026-10-02). On that base, with -Denable_cuda=true -Denable_float=true -Denable_checkasm=true, meson test gives 28 of 29 (the failure is test_cuda_pic_preallocation, which also fails on unpatched 9e48141b, 27 of 28 there), and the CPU -Db_sanitize=address,undefined -Db_lto=false build gives 22 of 25 on master and here (test_predict and test_pic_preallocation abort under LeakSanitizer and checkasm aborts on a heap-buffer-overflow in adm_dwt2_16 (integer_adm.c:2603); all three also abort on unpatched 9e48141b). The CLI output of the three Netflix pairs (--gpumask 0, vmaf_v0.6.1) against unpatched 9e48141b: src01 changes as described above (pooled VMAF 76.668905 to 76.66783, largest per-frame change 2.852e-3), both checkerboard pairs are identical (35.068667 and 7.985899). The sweeps and the other measurements below were taken on 8e7a1ac4e and not repeated; the files and x86 code paths they depend on are unchanged since (the one upstream change in between is an arm64-only ADM kernel).

• meson test: 27 of 28 pass on master, 28 of 29 here (the new test). The failure on both is test_cuda_pic_preallocation, which segfaults in the host-pinned case on unpatched master; this change does not touch it.
• CUDA CLI output (--gpumask 0, vmaf_v0.6.1) on the three Netflix pairs against master. The src01 pair changes by design: pooled VMAF 76.668905 to 76.66783 (the CPU gives 76.667831), largest per-frame change 2.9e-3. Both checkerboard pairs are identical (35.068667 and 7.985899). CPU output cannot change: no CPU file is touched, and no Python golden assertion changes.
• Not run: an ASan/UBSan build, 12 and 16-bit input.

Overlap: #1582 edits the same function (if (sup == 1) return 0; for tiny frames); git merge-file on motion_score.cu merges the two without conflict, and #1582 itself currently conflicts with master in integer_motion.c. #1614, #1553, #1613 and #1619 add tests to libvmaf/test/meson.build, #1583 and #1612 edit integer_motion_cuda.c: git merge-tree against each of them reports no conflict.

The workflow run on this PR needs a maintainer's approval.