feature/adm/cuda: clamp the scale 0 contrast masking neighbours at the border

[lusoris]

integer_adm_scale0 of adm_cuda differs from the CPU by up to 7.8e-2 on frames of 17 to 28 pixels once the row rounding is fixed (previous change in the series), and the kernel reads outside the band at those sizes.

This change applies on top of the row rounding change: that one removes the rounding error that otherwise hides this one, and the branch is stacked on it.

Cause

adm_cm_line_kernel() (libvmaf/src/feature/cuda/integer_adm/adm_cm.cu) reads the neighbours of a sample at the positions abs(x - 1), x and x + 1, and subtracts max(0, 2 * (x - w) + 1) from the last one; rows likewise, with y, the first row of the thread. The correction tests the sample's own position against the band, not the neighbour's, so it never applies: x is inside the band. For a scale 0 band of 14 samples or fewer in a dimension (a frame of 17 to 28 pixels) left and top are 0, the contrast masking runs to the band's last column and row, and the neighbour at x + 1 is column w (row h), outside the band. The CPU replicates the last sample there, {w - 2, w - 1, w - 1} (ADM_CM_THRESH_S_*, integer_adm.c).

The rows of a thread that lie below the band are masked off, so they do not change the score, but they read past the band as well.

Reproducer

Master 8e7a1ac4e, -Denable_cuda=true, RTX 4090, C API program printing %.17g (the new test makes the same comparison). A structured 20x20 8-bit frame (generator of the test), integer_adm_scale0:

	integer_adm_scale0
scalar CPU	0.609535
adm_cuda, master	0.907167
adm_cuda, row rounding change only	0.680597
adm_cuda, this branch on top of it	0.609535

Sweep of 212 configurations (17x17 to 64x64 and fifteen other sizes up to 320x180, 8 and 10 bits, random and structured content, two frames), largest absolute difference of integer_adm_scale0 against the scalar CPU over the configurations of the size group:

frame size	master	row rounding change	plus this change
17..28 square	3.2e-1	7.8e-2	0
17x64 and 64x17	4.1e-1	6.7e-2	0
29x29 and larger	see the row rounding change		unchanged

Rows (size, bit depth) with a scale 0 difference: 30 with the row rounding change alone, 2 with this change on top (55x55 and 320x180 at 8 bits, the angle test of the decouple stage, a separate change).

Scale 0 of the CPU is the same on master and with the fixes of #1599, #1600, #1602 and #1633 at every size of the sweep (checked on all 212 configurations), so the test needs none of them.

Fix

Both neighbours are clamped with min(position, n - 1): pos_x[0] = min(abs(x - 1), w - 1), pos_x[2] = min(x + 1, w - 1), and the same for the rows. For a sample inside the band that is the old position at the top and left (mirror), and the replicated last sample at the bottom and right.

Tests

test_cuda_adm_cm_scale0_border (new, libvmaf/test/, built with enable_cuda; it reports a skip when no CUDA device can be opened) compares integer_adm_scale0 of adm_cuda with the scalar CPU adm on 17x17, 20x20, 24x24, 28x28, 17x64 and 64x17 at 8 and 10 bits, random and structured content, with a tolerance of 1e-9. Unpatched master fails it (17x17, 8 bit, random: 1.8e-3), the row rounding change alone fails it too (1.81e-3 on the same case), this branch passes it.

Validation

Master 8e7a1ac4e, 2026-10-01. x86-64 Linux, GCC 16.2.1, nvcc 13.4, RTX 4090.

Rebased on master 9e48141b (2026-10-02). On that base, with -Denable_cuda=true -Denable_float=true -Denable_checkasm=true, meson test gives 29 of 30 (the failure is test_cuda_pic_preallocation, which also fails on unpatched 9e48141b, 27 of 28 there), and the CPU -Db_sanitize=address,undefined -Db_lto=false build gives 22 of 25 on master and here (test_predict and test_pic_preallocation abort under LeakSanitizer and checkasm aborts on a heap-buffer-overflow in adm_dwt2_16 (integer_adm.c:2603); all three also abort on unpatched 9e48141b). The CLI output of the three Netflix pairs (--gpumask 0, vmaf_v0.6.1) against unpatched 9e48141b: identical on all three pairs (src01 76.668905, checkerboard 1 px 35.068667, 10 px 7.985899). The sweeps and the other measurements below were taken on 8e7a1ac4e and not repeated; the files and x86 code paths they depend on are unchanged since (the one upstream change in between is an arm64-only ADM kernel).

• Real pairs, this branch (with the row rounding change) against master and against the scalar CPU, adm_cuda at %.17g (integer_adm2, scales 0 to 3, numerators and denominators): src01 576x324 8 bit (48 frames) and 10 bit (3 frames), checkerboard 1 px and 10 px 1920x1080 (3 frames each), KristenAndSara 1280x720, akiyo 352x288, sparks 480x270 10 bit (5 frames): identical to both. On the 18x22 akiyo clip the largest scale 0 difference to the CPU goes from 2.4e-2 (row rounding change alone) to 0 (all five branches merged; scales 1 to 3 close in the border change). No golden assertion changes.
• compute-sanitizer --tool memcheck reports no error for master and for the merge of the five branches at 24x24 and 20x36: the rows and columns past the band are inside the allocation.
• Merging with the other branches of the series: libvmaf/test/meson.build conflicts textually. cuda: keep kernel parameter structs and accumulators out of local memory #1595 and cuda: build the kernels with clang without the CUDA toolkit headers #1596 change adm_cm.cu lines that the row rounding change removes.
• Full meson test on the merge of the five branches: 28 of 29 pass. test_cuda_pic_preallocation aborts with SIGSEGV in test_cuda_picture_preallocation_method_host_pinned; it does the same on unpatched master.

The workflow run on this PR needs a maintainer's approval.