feature/adm: truncate the gain-limited sample in the AVX2 and AVX-512 decouple kernels

[lusoris]

With a fractional adm_enhn_gain_limit the AVX2 and AVX-512 integer ADM paths return a different integer_adm_scale0 from the scalar path. The default limit (100) and the limit of 1 that the vmaf_v1.0.16 models use are not affected, because the product is integral there.

Cause

adm_decouple() (libvmaf/src/feature/integer_adm.c) stores MIN(rst * adm_enhn_gain_limit, t) or MAX(...), a double, in an integer. That conversion truncates toward zero. Three vector kernels convert the product with an instruction that rounds to nearest:

• adm_decouple_avx2() (scale 0): _mm256_cvtpd_epi32
• adm_decouple_avx512() (scale 0): _mm512_cvtpd_epi32
• adm_decouple_s123_avx512() (scales 1 to 3): _mm512_cvtpd_epi64

They differ from the scalar kernel by one in every limited sample whose product has a fraction of one half or more. adm_decouple_s123_avx2() casts in C and already truncates.

A second defect sits in the same two scale 0 kernels. The angle test forms the dot product and the squared magnitudes with madd_epi16. The sum of two products of -32768 * -32768 is 2^31, which does not fit int32 and reads INT32_MIN, so a distorted sample with h = v = -32768 sets the angle flag where the scalar kernel, which sums in int64, clears it (the reference side forms the same sum). Decoded pictures cannot reach this: scale 0 coefficients stay within 22916. check_adm and any other caller that fills the bands directly can.

Reproducer

Master 9e48141b (reproduced there; the x86 code is unchanged since 6ec23e8), release build, AVX-512 host (--cpumask 16 is AVX2, -1 is scalar). Checkerboard pair, integer_adm_scale0 of the three frames:

for mask in -1 16 0; do
  build-release/tools/vmaf -r checkerboard_1920_1080_10_3_0_0.yuv -d checkerboard_1920_1080_10_3_1_0.yuv \
    -w 1920 -h 1080 -p 420 -b 8 --feature adm=adm_enhn_gain_limit=1.5 --no_prediction \
    --cpumask $mask -q --json -o adm_$mask.json
done

dispatch	master	this branch
scalar (-1)	0.665812, 0.605206, 0.665812	0.665812, 0.605206, 0.665812
AVX2 (16)	0.665846, 0.605206, 0.665846	0.665812, 0.605206, 0.665812
AVX-512 (0)	0.665846, 0.605206, 0.665846	0.665812, 0.605206, 0.665812

The CLI prints six digits, so the full comparison below used a throwaway copy of the tree with %.6f replaced by %.17g in output.c (not part of this change). Per-frame integer_adm_scale0, integer_adm_scale1 to 3 and integer_adm2, AVX2 and AVX-512 against scalar, frames that differ / frames, and the largest absolute difference:

pair	limit	master AVX2	master AVX-512	this branch
src01 576x324	1.0	0/48	0/48	0/48
	1.2	48/48, scale0 1.2e-6	48/48, scale0 1.1e-6	0/48
	1.5	45/48, scale0 3.1e-7	43/48, scale0 3.1e-7	0/48
	100	0/48	0/48	0/48
checkerboard 1 px	1.0, 1.2, 100	0/3	0/3	0/3
	1.5	2/3, scale0 3.4e-5	2/3, scale0 3.4e-5	0/3
checkerboard 10 px	all four	0/3	0/3	0/3

integer_adm_scale1 to 3 do not differ on these decoded pairs even with the AVX-512 scale 1 to 3 kernel; the new test shows that kernel's difference on bands that take the limit (below).

Fix

The three conversions become _mm256_cvttpd_epi32, _mm512_cvttpd_epi32 and _mm512_cvttpd_epi64. t is an integer, so truncating the product and then taking the integer minimum or maximum gives the scalar's sample. In the scale 0 angle test the squared magnitudes are read as unsigned (uint32_t lanes in AVX2, _mm512_cvtepu32_ps in AVX-512) and a dot product of INT32_MIN is mapped back to 2^31.

Tests

• test_adm_decouple_gain (new, libvmaf/test/) runs the scale 0 kernels and the AVX-512 scale 1 to 3 kernel against the scalar kernels at limits 1, 1.2, 1.5 and 100, on bands in which three samples in four are a scaled copy of the reference (so the angle test passes and the limit is taken), and the scale 0 kernels again on bands that reach -32768. The first case uses the scale 0 coefficient bound of 22916 so that it isolates the rounding. On unpatched master it fails: AVX2 differs in 710 to 2470 samples per frame at limit 1.2 and AVX-512 in 492 to 2138 (scale 0), 562 to 2248 (scale 1 to 3). With only the conversions fixed, the -32768 case still fails (36 samples at 64x48, limit 1.2). The AVX2 scale 1 to 3 kernel is not compared: it passes lanes below 2^15 to get_best15_from32, a negative shift under UBSan.
• check_adm's decouple test keeps the sizes, patterns and gains that adm: add NEON scale-zero decoupling #1656 added for the AArch64 NEON kernel and adds, on every architecture, the limits 1.2 and 1.5 and a pattern (5) whose distorted bands are derived from the reference (independent random bands almost never pass the angle test); x86 runs patterns 0 and 5. It also calls div_lookup_generator(): div_lookup is a static array in integer_adm.h, and only adm_buffer_alloc() in integer_adm.c fills its own copy, so the copy in check_adm.c was never filled. With master's adm_avx2.c and adm_avx512.c and this test, checkasm --test=adm fails 50 to 72 of 415 tests on each of 20 seeds. On this branch it passes on the same 20 seeds (415 tests).
• After adm: add NEON scale-zero decoupling #1656: upstream's NEON cases are kept as they are. With the real lookup, the limited-sample branch of adm_decouple_neon() now runs in checkasm; with the zero lookup the reconstruction ratio was zero and it never did. On an aarch64 cross build under qemu-user checkasm --test=adm passes with upstream's kernel (507 tests); a kernel that adds 1 to the limited sample fails it (55 of 507), and one that rounds the reconstruction instead of truncating fails 114 of 507. A shifted angle threshold (x1.0000001) or > for >= 0 in the angle test is not caught; the NEON kernel itself is unchanged by this PR.
• Planted defects: reverting only the AVX-512 scale 1 to 3 conversion, or only the AVX-512 scale 0 conversion, fails the new test.

Validation

Rebased on master 9e48141 (2026-10-02). x86-64 Linux, GCC 16.2.1, on master 9e48141b. The checkasm --bench cycle counts in this description were measured on 6ec23e8 and not repeated; the %.17g comparison table, the test counts, checkasm --test=adm and the reference-pair comparison were re-run on 9e48141.

• Release build meson test with -Denable_checkasm=true: 25/25 on master, 26/26 here (the new test), including checkasm.
• -Db_sanitize=address,undefined build: 22 pass on master, 23 pass here (checkasm is one of the three that abort, in the existing adm_dwt2_16 test). test_predict and test_pic_preallocation abort on LeakSanitizer reports on both; this change does not touch them.
• The three Netflix reference pairs (.engagement/golden_compare.py): identical per-frame and pooled output between master and this branch, default dispatch and --cpumask -1. Also identical at %.17g for --cpumask 0, 16 and -1. No golden assertion changes.
• Cost of the decouple stage, checkasm --bench, limit 100, cycles per call, median of 7 alternating runs (Ryzen 9 9950X3D): within run-to-run noise. The unchanged scalar kernel moves by up to 13% between the two builds, the vector kernels by 0.3% to 10%, none consistently in one direction (adm_decouple_64x48: AVX2 14019 to 13860, AVX-512 11766 to 11726; adm_decouple_s123_64x48: AVX2 28056 to 27345, AVX-512 18596 to 18618).

Overlap: #1602 and #1600 also touch adm_avx2.c, adm_avx512.c, check_adm.c and libvmaf/test/meson.build. git merge-tree against both branches reports no conflict.

The workflow run on this PR needs a maintainer's approval.

[lusoris]

Rebased onto acdd937; the new head is 372ca14. The x86 kernels did not change upstream, so the fix is the same. The conflict was libvmaf/test/checkasm/check_adm.c, in check_adm_decouple_s123(): upstream's NEON commits added aarch64 sizes, gains (1.0, 1.1, 100) and value patterns to that test, which this PR also edits. I kept upstream's structure and its aarch64 cases and added to it: the gains 1.2 and 1.5 (so rst * gain is fractional) on every architecture, a pattern 5 whose distorted bands are scaled copies of the reference (so the angle test passes and the limit is taken), run on x86 as well, and the div_lookup generator.

Release build with checkasm: all 28 meson tests pass, test_adm_decouple_gain passes, and checkasm --test=adm passes for seeds 1 to 5 (445 tests). With adm_avx2.c and adm_avx512.c put back to master's, test_adm_decouple_gain fails and checkasm reports 51 of 445 failures, so both catch the bug. ASan/UBSan build: 24 pass and 4 fail (test_predict, checkasm, test_read_pictures_convert, test_pic_preallocation); the same 4 fail on unpatched master acdd937. I did not run aarch64 or an AVX-512 host.