fix(core): sanitizer-pass cleanup — CAMBI option type-mismatch + ADM signed-shift UB

[lusoris]

Summary

Local -Db_sanitize=address,undefined audit on master tip bbcaa8d127
surfaced two real undefined-behaviour findings that the per-PR CI
sanitizer combination is not currently catching. Both are bit-exact
fixes — no numerical-correctness delta.

1. CAMBI option-parser misaligned uint16_t write. Options table
   declared window_size and max_log_contrast as VMAF_OPT_TYPE_INT
   but the underlying CambiState fields were uint16_t. The
   parser's *(int *)data = value (a) silently clobbered adjacent
   src_window_size when writing window_size and (b) hit a
   UBSan-flagged misaligned 4-byte store on max_log_contrast
   (2-byte-aligned slot). Mirror read-side misalignment in
   vmaf_feature_name_from_options (core/src/feature/feature_name.c:104)
   per frame.

2. AVX2 / AVX-512 ADM DWT2 filter packing — left-shift of negative
   signed int. (uint32_t)(filter[k] << 16) casts the result, not
   the operand — the inner shift runs on a signed int. UB on every
   HBD ADM frame; UBSan reports left shift of negative value -4240.

Type

• fix — bug fix
• simd — backend-specific (AVX2 / AVX-512 ADM)

Checklist

• Conventional Commits.
• pre-commit run --files is green locally on every touched file
  (clang-format, secrets, ADR-0332, ADR-0105, ADR-0386 checks all
  pass; semgrep clean on the C diff).
• Unit tests pass: meson test -C build-asan — 63/63 OK with ASan
  + UBSan + leak detection enabled.
• Touched SIMD code paths (AVX2 + AVX-512 ADM): bit-exact fix
  (cast move inside parentheses); /cross-backend-diff style
  worst-ULP check unchanged. test_integer_adm_simd still
  passes — scalar / AVX2 / AVX-512 ULP 0.
• All SIMD/GPU twins of the touched extractor: the AVX2 + AVX-512
  twins both fixed in this PR (only x86 twins existed for ADM
  DWT2 filter packing).
• No new files require licence headers (C edits are in existing
  files; the new docs/changelog files are markdown).
• Not a breaking change.

Bug-status hygiene (ADR-0165)

• docs/state.md updated — new row
  T-SANITIZER-PASS-CLEANUP-2026-05-30 in ## Recently closed.

Netflix golden-data gate (ADR-0024)

• No assertAlmostEqual(...) golden assertions modified. Both
  fixes are bit-exact with prior behaviour on every input.
• No CODEOWNERS exception needed.

Cross-backend numerical results

adm   cpu-vs-avx2-ULP=0   cpu-vs-avx512-ULP=0   (preserved; bit-exact)
cambi cpu-only            no GPU twins for this fix path

Performance

Not a perf change. The CAMBI fix adds two assignments and 8 bytes per
extractor instance (one per pipeline). The ADM fix is one character
per call site and produces identical machine code under any optimising
compiler with two's-complement integer representation.

Deep-dive deliverables (ADR-0108)

• Research digest — docs/research/sanitizer-pass-2026-05-30.md.
• Decision matrix — see docs/adr/0869-sanitizer-pass-cleanup.md §Alternatives considered (5 rows: option-type-add, struct-type-flip, shadow-int-bridge chosen, NOLINT-and-ignore, branch-on-sign).
• AGENTS.md invariant note — no rebase-sensitive invariants: this PR fixes a 4-character cast move + a struct shadow slot; not invariant-class changes.
• Reproducer / smoke-test command — pasted below.
• CHANGELOG fragment — changelog.d/fixed/sanitizer-pass-cleanup.md.
• Rebase note — entry added to docs/rebase-notes.md under sanitizer-pass-cleanup (2026-05-30, ADR-0869).

Reproducer

# Reproduce the original UBSan misalignment + signed-shift on master:
git checkout origin/master
meson setup build-asan core -Denable_cuda=false -Denable_sycl=false \
  -Db_sanitize=address,undefined
ninja -C build-asan

# Cambi misaligned-store (UBSan-flagged on max_log_contrast):
ASAN_OPTIONS=halt_on_error=0 UBSAN_OPTIONS=print_stacktrace=1:halt_on_error=0 \
  ./build-asan/tools/vmaf \
    --reference python/test/resource/yuv/src01_hrc00_576x324.yuv \
    --distorted python/test/resource/yuv/src01_hrc01_576x324.yuv \
    --width 576 --height 324 --pixel_format 420 --bitdepth 8 \
    --feature cambi
# expected on master: 2 runtime errors (store + load misalignment)
# expected on this PR: silent

# ADM signed-shift UB (HBD path):
./build-asan/tools/vmaf \
  --reference python/test/resource/yuv/src01_hrc00_576x324.yuv422p10le.yuv \
  --distorted python/test/resource/yuv/src01_hrc01_576x324.yuv422p10le.yuv \
  --width 576 --height 324 --pixel_format 422 --bitdepth 10 \
  --feature adm
# expected on master: "left shift of negative value -4240" per frame
# expected on this PR: silent

Findings detail

#	File	Finding	Root cause	Severity
1	core/src/feature/cambi.c (CambiState)	UBSan misaligned 4-byte store + load + silent struct-field overwrite	VMAF_OPT_TYPE_INT parser writes into uint16_t slot at 2-byte-aligned offset; clobbers adjacent field on window_size	UB + silent struct corruption — triggered on every --feature cambi
2	core/src/feature/x86/adm_avx{2,512}.c (4 sites each)	UBSan left shift of negative value	Cast on shift result, not operand; signed shift of negative int is UB	UB — triggered on every HBD ADM frame

Known follow-ups

• CambiState::enc_width, enc_height, enc_bitdepth, src_width,
  src_height are unsigned but targeted by VMAF_OPT_TYPE_INT.
  Layout-compatible on every shipped ABI; UBSan does not flag. A future
  stricter option-type schema would discharge the class.
• AdmState::adm_adm3_apply_hm is int but targeted by
  VMAF_OPT_TYPE_BOOL. Safe by accident
  (memset(priv, 0, priv_size) + 1-byte bool write). Same class.

Both noted in ADR-0869 §Consequences as follow-up audit candidates.

[lusoris]

Watch-loop skipped: rebase onto current origin/master produced multiple add/add conflicts including python/test/feature_extractor_test.py, quality_runner_test.py, routine_test.py (Netflix golden-data territory — agent may not touch per CLAUDE §8/§12 r1) plus docs/state.md, docs/rebase-notes.md, docs/adr/_index_fragments/_order.txt, core/test/meson.build. Manual rebase required. Loop continues to next queue PR.

[lusoris]

Recovered and reopened. The original branch was force-pushed to master tip (bc0f002f1b) at 2026-05-30T22:50:24Z when the watch-loop mis-handled the rebase-conflict at 21:25 — the underlying fix-content was lost from the branch but the orphan commit 6270309d still exists in the repo.

This force-push restores the actual fix on top of current origin/master (bc0f002f1b):

• C-source diff is unchanged (cambi.c +24/-2, adm_avx2.c +16/-4, adm_avx512.c +16/-4).
• Doc deliverables (ADR-0869, research digest, changelog, state.md row, rebase-notes entry, _order.txt, README index row, AGENTS.md invariant) all reconstructed.
• Now applies cleanly without conflicts because it's properly rooted at master.

Kept as DRAFT — train-watcher will promote when ready.