feature/adm: keep get_best15_from32() inside its defined range on AVX2

[lusoris]

adm_decouple_s123_avx2() calls get_best15_from32() on every lane, including lanes below 2^15 whose result it throws away. For those lanes the function shifts by a negative count, and for a value of 0 it calls __builtin_clz(0). UBSan reports both. Scores do not change.

Cause

The scalar adm_decouple_s123() and the AVX-512 kernel call the function only for values of 32768 and above:

uint16_t kh_msb = (abs_oh < (32768) ? abs_oh : get_best15_from32(abs_oh, &kh_shift));

The AVX2 kernel cannot branch per lane. It extracts all eight lanes, runs get_best15_from32() on each, and selects afterwards with _mm256_cmpgt_epi32(const_32768_epi32, abs_oh_epi32) (kh, kv and kd alike). Below 32768 the shift count 17 - __builtin_clz(temp) is zero or negative, so (1 << (k - 1)) and >> k are undefined, and temp == 0 has no defined __builtin_clz. The 24 per-lane calls in adm_decouple_s123_avx2() are the AVX2 callers that see such values; the three calls in the scalar tail of the same function (about line 1883) already guard with < 32768.

Reproducer

Master 6ec23e8f2 (reproduced again on 9e48141b: 3 UBSan reports on master, 0 with this patch), ASan/UBSan build. --cpumask 16 keeps AVX-512 off so that the AVX2 kernel runs on an AVX-512 machine; UBSan reports each site once:

meson setup build-san libvmaf --buildtype debug -Db_sanitize=address,undefined -Db_lto=false
ninja -C build-san
build-san/tools/vmaf -r src01_hrc00_576x324.yuv -d src01_hrc01_576x324.yuv -w 576 -h 324 -p 420 -b 8 \
    --feature adm --cpumask 16 --threads 1 -q --json -o out.json

On master:

adm_avx2.c:1348:13: runtime error: passing zero to __builtin_clz(), which is not a valid argument
adm_avx2.c:1350:23: runtime error: shift exponent -1 is negative
    #0 get_best15_from32 libvmaf/src/feature/x86/adm_avx2.c:1350
    #1 adm_decouple_s123_avx2 libvmaf/src/feature/x86/adm_avx2.c:1543
    #2 integer_compute_adm libvmaf/src/feature/integer_adm.c:2954
adm_avx2.c:1350:36: runtime error: shift exponent -4 is negative

On this branch the same command prints no runtime error. This is the report that #1599 and #1601 list as present before and after their change; it also appears as the remaining report in the frame-size sweep of #1599.

Fix

get_best15_from32() in adm_avx2.c returns, for temp < 32768, the value itself and a shift of 0:

if (temp < 32768) {
    *x = 0;
    return (uint16_t) temp;
}

That is what the caller's select picks for those lanes: blend(abs_oh_epi32, tmp_kh_msb_epi32, mask) takes abs_oh where the mask is set (abs_oh < 32768), and blend(_mm256_setzero_si256(), kh_shift_epi32, mask) takes a shift of 0. The discarded lanes carry the same bits as before, so no result moves. Values of 32768 and above take the old path.

Tests

No new test. The defect is undefined behaviour whose result is discarded: on x86 the negative shift and the clz(0) produce a value that the select throws away, so no output differs and a test cannot fail on master without a sanitizer. check_adm's adm_decouple_s123 test already covers this kernel for equality with the scalar kernel and passes on master and here; under UBSan it reports the same three errors on master (checkasm --test=adm --function='adm_decouple_s123*' in the ASan/UBSan build: 3 reports on master, 0 here). The UBSan reproducer above is the regression check. (The full checkasm --test=adm aborts under ASan on master and here with a heap-buffer-overflow in adm_dwt2_16 (called from check_adm.c:382), unrelated to this change, so the filter is needed; under ASan the filtered run also reports "missing vzeroupper" for both AVX2 and AVX-512 on master and here, which the release build does not.)

Validation

Rebased on master 9e48141 (2026-10-02). x86-64 Linux, GCC 16.2.1, on master 9e48141b, Ryzen 9 9950X3D, -Denable_float=true. The %.17g comparison and the checkasm --bench cycle counts below were measured on 6ec23e8 and not repeated; the UBSan reproducer, the test counts, checkasm --test=adm and the reference-pair comparison were re-run on 9e48141.

• Release build meson test (-Denable_float=true -Denable_checkasm=true): 25/25 on master, 25/25 here. -Db_sanitize=address,undefined -Db_lto=false build: 22 pass and 3 fail on master, 22 pass and 3 fail here; test_predict and test_pic_preallocation fail on LeakSanitizer reports and checkasm aborts on a heap-buffer-overflow in adm_dwt2_16 (integer_adm.c:2603 on master), on both, and this change does not touch them.
• checkasm --test=adm (release, -Denable_checkasm=true): all 265 tests pass on master and here.
• The three Netflix reference pairs (.engagement/golden_compare.py): identical per-frame and pooled output between master and this branch, default dispatch and --cpumask -1. No golden assertion changes.
• %.17g comparison through the C API at --cpumask 16, 0 and -1 on the three pairs: vmaf (vmaf_v0.6.1), adm2, aim, adm3, the four ADM scale scores, vif scale 0 and motion2, 480 values for src01 and 30 for each checkerboard pair per mask, all bit-identical between master and this branch (9 of 9 pair/mask combinations).
• Cost, checkasm --bench of adm_decouple_s123, cycles per call, median of 8 alternating runs pinned to one core. The AVX2 kernel gets faster, because the guarded lanes skip the clz and shifts: 16x16 2237 to 1929, 32x20 5720 to 4723, 33x21 6398 to 5582, 64x48 25388 to 22357, 65x49 26935 to 22711 (11.9% to 17.4% fewer cycles). The AVX-512 kernel is unchanged within noise (-1.0% to +2.3%). The unchanged scalar kernel moves by -0.9% to +2.0% between the two builds, which is the noise level. Three bench runs stalled (killed after 90 s or by hand) and were dropped; the numbers above come from the 8 completed runs per build.

Overlap: git merge-tree of this branch against the branches of #1599, #1600, #1601, #1602, #1603, #1631, #1633 and #1634 reports no conflict with any of them (#1599, #1600, #1601, #1602 and #1633 touch adm_avx2.c or check_adm.c; their hunks do not meet this one).

The workflow run on this PR needs a maintainer's approval.

[lusoris]

Correction on my side: this is the same fix as #1605, which I closed on 2026-09-21 after @anematode pointed out that #1584 removes this code path (it replaces the per-lane get_best15_from32() calls with a vector helper). I reopened it today from the old branch without checking that history.

The overlap, so nobody reviews this twice:

• avx2: vectorize a couple scalar parts #1584 (open, last updated 2026-09-21, merges cleanly onto 6ec23e8): rewrites the 24 per-lane calls in adm_decouple_s123_avx2(). With it merged, this PR has nothing left to fix.
• This PR: nine lines in get_best15_from32() itself, no change to the kernel.

Either one resolves the UBSan reports in the description. If #1584 lands, close this. I am leaving it open only as the smaller change in case #1584 is not ready.