fix(core): clear *pool on init failure to prevent UAF in vmaf_gpu_picture_pool_init

[lusoris]

Summary

• Fixes a high-severity use-after-free in vmaf_gpu_picture_pool_init() (core/src/gpu_picture_pool.c) where the combined assignment VmafGpuPicturePool *const p = *pool = malloc(...) publishes the pool pointer to the caller's *pool before any later failure path runs. On goto free_p the function frees p while *pool still holds the dangling pointer; the CUDA call site (libvmaf.c:326) stores that handle in the long-lived VmafContext.cuda.ring_buffer, and vmaf_close() then calls vmaf_gpu_picture_pool_close() on the freed object — UAF + potential double-free.
• Fix is mechanical: set *pool = NULL after the !p malloc check and after free(p) at the free_p label. The fail: label is unchanged.
• Adds a CPU-only test_gpu_picture_pool_uaf regression in suite=fast that exercises the goto free_p arm by passing a pic_cnt large enough to make the second malloc() return NULL, then asserts *pool == NULL.
• Documents the invariant in core/src/AGENTS.md (chore(deps): Update rocm/rocm-terminal Docker tag to v6.4 #10) so future out-parameter init functions inherit the clear-on-failure contract by convention.

Test plan

• meson setup build-cpu core -Denable_cuda=false -Denable_sycl=false -Denable_hip=false
• ninja -C build-cpu — green
• meson test -C build-cpu test_gpu_picture_pool_uaf --print-errorlogs — pass
• meson test -C build-cpu --suite=fast — 50/50 pass
• Regression verification: with the *pool = NULL after free(p) reverted, the new test fails on the "pool handle must be NULL after init failure (UAF regression)" assertion. With the fix in place it passes in <1 ms.
• clang-tidy -p build-cpu core/src/gpu_picture_pool.c core/test/test_gpu_picture_pool_uaf.c — clean
• clang-format --dry-run --Werror on touched C files — clean
• CI required-checks aggregator (will fire on ready-for-review).

ADR-0108 deep-dive deliverables

Research digest: [x] no research digest needed: small targeted bug fix per CLAUDE.md §12 r11 ("trivial deliverable" sentinel). Root cause + fix + the surrounding ring-buffer call site are explained inline in the commit body and pinned as core/src/AGENTS.md invariant #10.

Decision matrix: [x] no alternatives: only-one-way fix. Clearing *pool = NULL at the failure labels is the minimal correctness-preserving change. Hoisting the *pool = ... assignment to the success tail was considered but rejected — it changes the emission shape of the const-init pattern reviewers grep for, and would still need a separate NULL-init for the early failure case.

AGENTS.md invariant note: core/src/AGENTS.md invariant #10 added — "Out-parameter init functions must clear the caller's handle on every failure path." Names vmaf_gpu_picture_pool_init as the exemplar and points to the regression test.

Reproducer / smoke-test command:

meson setup build-cpu core -Denable_cuda=false -Denable_sycl=false -Denable_hip=false
ninja -C build-cpu
meson test -C build-cpu test_gpu_picture_pool_uaf --print-errorlogs

Without the fix the test fails on the "pool handle must be NULL after init failure (UAF regression)" assertion. With the fix it passes in microseconds.

CHANGELOG fragment: changelog.d/fixed/gpu-picture-pool-uaf-init-failure.md (per ADR-0221).

Rebase note: docs/rebase-notes.md — gpu-picture-pool-uaf-on-init-failure (2026-05-30) entry added. The file core/src/gpu_picture_pool.c is fork-added by ADR-0239 (promotion of upstream's cuda/ring_buffer.c into a backend-agnostic helper); upstream Netflix/vmaf still ships the original cuda/ring_buffer.c, so any upstream parallel fix will land in a different TU and won't textually conflict. docs/state.md updated under "Recently closed" with the T-GPU-PICTURE-POOL-UAF-INIT-FAILURE-2026-05-30 row per CLAUDE.md §12 r13.

🤖 Generated with Claude Code

[lusoris]

Closing as part of marathon cleanup 2026-05-31 (150 PRs merged today). Content likely superseded by sibling merges. Reopen if specific finding still needs work; bigger PRs preferred going forward per session feedback.

[lusoris]

Superseded by #515 — bundled into one PR per bigger-PRs guidance. All three diffs preserved verbatim.