fix(adm): form the 16-bit vertical DWT sum in int64

[lusoris]

Summary

Integer ADM's 16-bit vertical DWT pass summed four weighted samples in int32_t. That overflows on bright 16-bit input, which is undefined behaviour, in the scalar path and in both x86 kernels. The sum is now formed in 64 bits. No score changes. Stacked on #1473, parallel to #1474 and #1476; review #1473 first.

The defect. Scale 0 forms sum(filter[k] * s[k]) and only then subtracts 46342 * 2^(bpc - 1). The first three low-pass taps (15826, 27411, 7345) add up to 50582, so at 16 bpc the partial sum passes INT32_MAX once three consecutive samples reach 42456. The same int32 sum sits in three places:

• scalar adm_dwt2_vpass_16();
• the scalar vertical loop of adm_dwt2_16_avx2();
• the scalar vertical loop of adm_dwt2_16_avx512().

Upstream Netflix/vmaf has the same code.

Why nothing noticed. The normalised value is at most 54822 * 32768 in magnitude, inside int32, so two's-complement wrap-around undoes the overflow and every path gets the right answer on the hardware we test. Parity tests cannot see it; only a sanitizer build can.

The fix. A new adm_dwt2_vpass16_tap4() in integer_adm.h forms the response in int64, as the file's i4_dwt2_tap4() already does for scales 1–3. All three kernels share it. For in-range input the result equals the old wrapped one. The 8-bit pass is unchanged: 255 × 50582 fits in int32.

Regression test. test_integer_adm_dwt16_range scores bright 16-bit noise at two sizes on every dispatch level and requires SIMD to equal scalar bit for bit. The sanitizer lane (halt_on_error=1) is what gives it teeth: under UBSan it reaches all nine overflow sites without the fix, three per kernel, and none with it.

no docs needed: bug fix with no user-visible change; every score is byte-identical before and after, and no option, output or supported input changes.

Type

• feat — new feature
• fix — bug fix
• perf — performance improvement
• refactor — no behavior change
• docs — documentation only
• test — test-only
• build / ci — tooling / infra
• port — cherry-pick from upstream Netflix/vmaf
• sycl / cuda / simd — backend-specific

Checklist

• Commits follow Conventional Commits (the commit-msg hook enforces this).
• make format && make lint is green locally: pre-commit on every changed file passes, and clang-tidy (CPU lane) and cppcheck report 0 findings on the four touched TUs.
• Unit tests pass: meson test -C build: fast suite 139/139.
• If I touched any SIMD/GPU code path, I ran /cross-backend-diff and the worst ULP is ≤ 2: 0 ULP. Scalar, AVX2 and AVX-512 are bit-identical at %.17g before and after, and to each other.
• If I touched a feature extractor with SIMD/GPU twins, I either updated every twin or listed the gap under "Known follow-ups" below: the CUDA, HIP and Metal twins carry the same int32 sum and are listed below (SYCL already uses int64). NEON has no 16-bit DWT.
• If I added a new .c / .cpp / .cu / .h / .hpp, it has the appropriate license header (see CONTRIBUTING.md).
• If this is a breaking change, the commit message uses ! or BREAKING CHANGE: and the migration path is documented below: not breaking.
• If this PR adds an ADR, the ADR row lives in docs/adr/_index_fragments/<NNNN-slug>.md and the slug is appended to docs/adr/_index_fragments/_order.txt: no ADR, bug fix.

Bug-status hygiene (ADR-0165)

• docs/state.md: T-ADM-DWT2-16BIT-INT32-OVERFLOW-2026-09-18 closed. T-GPU-ADM-DWT2-16BIT-INT32-OVERFLOW-2026-09-18 opened for the GPU twins.

Netflix golden-data gate (ADR-0024)

• I did not modify any assertAlmostEqual(...) score in the Netflix golden Python tests.
• If I believe a golden value must change, I have explained why below AND pinged @lusoris for a CODEOWNERS exception: no score changes, and the golden pairs are 8-bit, which this does not touch.

Deep-dive deliverables (ADR-0108)

• Research digest — docs/research/2063-upstream-sync-2026-09-adm-vif-simd.md, new section "Int32 overflow in the 16-bit vertical DWT pass".
• Decision matrix — no alternatives: only-one-way fix (no ADR; the three options weighed, int64 vs centring the samples vs uint32_t, are tabulated in the digest section).
• AGENTS.md invariant note — core/src/feature/AGENTS.md, "Integer ADM's 16-bit vertical DWT sums in int64".
• Reproducer / smoke-test command — below.
• CHANGELOG fragment — changelog.d/fixed/adm-dwt2-16bit-overflow.md.
• Rebase note — docs/rebase-notes.md: keep adm_dwt2_vpass16_tap4() when a sync touches the 16-bit vertical loops.

Reproducer

# UBSan build: the test aborts on the old int32 sum and passes on the fix
CC=clang CXX=clang++ meson setup build-ubsan core -Denable_cuda=false -Denable_sycl=false \
  -Db_lto=false -Db_sanitize=undefined -Db_lundef=false --buildtype=debugoptimized
ninja -C build-ubsan test/test_integer_adm_dwt16_range
UBSAN_OPTIONS=halt_on_error=1:print_stacktrace=1 build-ubsan/test/test_integer_adm_dwt16_range

# Scores unchanged: compare a release build before and after on any 16-bit clip
vmaf -r ref16.yuv -d dis16.yuv -w 176 -h 144 -p 420 -b 16 --feature adm --no_prediction \
  --cpumask 0 --precision max -o out.json --json    # also --cpumask 48 (AVX2) and 65535 (scalar)

Verified locally

Check	Result
test_integer_adm_dwt16_range, UBSan, old code	fails; nine reports (integer_adm.c 1463/1464/1513, adm_avx2.c 3392/3393/3397, adm_avx512.c 3495/3496/3500)
Same test, UBSan and release, with the fix	pass, no reports
Scores before vs after, --precision max	identical in 12 of 12 runs: 10, 12 and 16 bpc (two 16-bit clips) × scalar, AVX2, AVX-512
Fast suite (release build)	139/139
scripts/dev/preflight.sh	gcc, clang, msvcism, tidy, cppcheck pass. The ASan+UBSan stage passes 138/139, the new test included. The two failures are the pre-existing false positives #1475 fixes: check_exported_symbols flags ASan's __start_asan_globals under GNU ld, and m32 sweeps the x86/ and arm64/ files the i686 lane never builds
clang-tidy (CPU lane) + cppcheck on the touched TUs	0 findings, 0 uncited NOLINTs

Known follow-ups

• The CUDA (adm_dwt2.cu), HIP (adm_dwt2.hip) and Metal (integer_adm.metal) twins sum the same response in int32 (T-GPU-ADM-DWT2-16BIT-INT32-OVERFLOW-2026-09-18); the SYCL twin already forms it in int64. Their fix will be a PR stacked on fix(gpu): make the integer ADM twins agree with the CPU on tiny frames and full-range content #1476.
• Worth reporting upstream with the port: reconcile with upstream Netflix/vmaf September 2026 (ADM/VIF SIMD fixes) #1473 findings: Netflix/vmaf has the identical int32 sum.

[lusoris]

Absorbed into the ADM stack train #1507, per your direction to fold this stack the way #1506 was folded.

This PR targeted the one below it in a five-deep stack, so none of the five could merge until every one below had merged and been restacked — five sequential rebase-plus-CI rounds. #1507 is one. Your work is in it unchanged; that PR's description lists the six defects the fold itself surfaced, none of which an individual PR could see, because each gate only looks at the files its own PR touches.

The branch stays on the remote.