fix(core): vmaf_read_pictures owns its pictures on every return (ADR-1431)

[lusoris]

Summary

With the device's memory taken by another process, vmaf --backend cuda printed problem reading pictures and then never exited: it sat at 0 % CPU in vmaf_close(), holding whatever lock the caller held, until it was killed. A failed vmaf_read_pictures() kept the pair of pictures it was given; the pair came from the CLI's picture pool and never went back. The call now owns both pictures on every return, and the CLI exits with the error at once.

Closes T-READ-PICTURES-FAILURE-LEAKS-POOL-PICTURES-2026-10-01 (found here, ADR-1431). I found it checking Netflix/vmaf#1420, which reports that vmaf_cuda_buffer_alloc() asserts on an out-of-memory; the fork does not assert there (CHECK_CUDA returns the error, test_cuda_buffer_alloc_oom pins -ENOMEM). RC3, reliability (ADR-1421).

Cause

gdb on the hung CLI: the main thread waits in pthread_cond_wait in vmaf_close() -> vmaf_commit_remaining_owners() -> vmaf_picture_pool_close(), which returns only when every pool picture is back.

vmaf_read_pictures() returned without releasing its pictures when its validation step failed (a non-increasing index, pictures whose shape disagrees with the stream, check_picture_pool, check_ring_buffer where the OOM happens, the SYCL host-upload preparation) and when the CUDA translation failed. The failures after that point (context fallback, an extractor, the post-extractor stage) released them. docs/api said the caller keeps the pictures after any error, libvmaf.h said the context takes them, and every caller in the tree (vmaf, vmaf_bench, MCP compute_vmaf, vmaf_vpl, the libvmaf_tune filter) leaves them alone after an error: a caller that followed docs/api would have released a picture the extractor paths had already released.

What changed

• core/src/libvmaf.c: from the moment it has a context and two pictures, every return of vmaf_read_pictures() releases both once (read_pictures_frame_cleanup(); the new read_pictures_translate_abort() for the CUDA translations, which may share storage with the caller's pictures). No context, one picture NULL and the flush call take nothing. check_ring_buffer() returns the real error (-ENOMEM) instead of -EINVAL.
• docs/api/index.md and the vmaf_read_pictures() Doxygen state the rule. test_read_pictures_monotonic and test_validate_pic_params_bpc stop unref'ing after a rejection.

Output on an RTX 4090

hog (upstream's, holds all but 400 MiB of the device) and vmaf --backend cuda on a 4096x2160 pair, 2 frames:

	Before	After
Exit	killed after 28 minutes at 0.1 % CPU, device lock held	244 (-ENOMEM) in about 5 s, twice
Message	problem reading pictures, then nothing	libvmaf ERROR problem during prepare_ring_buffer, problem reading pictures

test_cuda_oom_pictures_released (takes the device's memory itself, submits a pair from a pool of one pair, expects the error, frees the memory, submits the same index again): passes twice with the fix (first-frame error -12); without it the second submit never returns and the 60 s alarm ends the run (exit 142). test_read_pictures_failure_ownership (CPU): fails on master (alarm), passes with the fix. python3 scripts/ci/run_meson_test.py -- -C bs --suite=fast on an ASan + UBSan build: 215 OK, 0 failed.

Not measured: a HIP run under VRAM pressure. The gfx1036 allocates from system memory (hipMalloc of 30 GiB of 31 GiB succeeded and vmaf --backend hip still ran), so the hog does not starve it; the HIP twins read their pictures inside their extractors, where the failure path already released them.

Type

• fix — bug fix

Deep-dive deliverables (ADR-0108)

• Research digest — no digest needed: the cause and the measurements are in the state row and ADR-1431.
• Decision matrix — ## Alternatives considered in ADR-1431.
• AGENTS.md invariant note — core/src/AGENTS.md, "vmaf_read_pictures() owns both pictures on every return".
• Reproducer / smoke-test command — under "Reproducer" below.
• CHANGELOG fragment — changelog.d/fixed/read-pictures-owns-pictures-on-failure.md.
• Rebase note — docs/rebase-notes.md, "ADR-1431 — vmaf_read_pictures() owns its pictures on every return".

Reproducer

# CPU, no device: the pool has no spare picture, the alarm turns a hang into a failure
build/test/test_read_pictures_failure_ownership
# CUDA host
flock ~/.cache/vmafx-locks/cuda-4090.lock timeout 120 build/test/test_cuda_oom_pictures_released

End to end: upstream's hog 400 40 & (/home/kilian/.cache/vmafx-upstream-rebase/evidence/c1420/hog.c), then vmaf -r z.yuv -d z.yuv --width 4096 --height 2160 --pixel_format 420 --bitdepth 8 --frame_cnt 2 --backend cuda -o o.json --json inside the device lock with a timeout.

Notes

• docs/api/index.md and libvmaf.h are also touched by the index-gap documentation PR; different lines.
• Touches core/test/meson.build (two blocks) and core/src/libvmaf.c (vmaf_read_pictures(), check_ring_buffer() and one new static function).
• No ABI or FFmpeg patch impact (callers already leave the pictures alone). No Netflix golden assertion changed.