fix(gpu): let the backend flush own GPU extractors under --threads (ADR-1197)

[lusoris]

Summary

vmaf --threads N failed on every GPU backend, for every N including 1, on every input:

libvmaf ERROR context could not be synchronized
exit 234

The message is wrong, and that is the first thing worth stating. Instrumenting flush_context_cuda() shows all four driver calls returning success — cuCtxPushCurrent=0 cuStreamSynchronize=0 cuCtxSynchronize=0 cuCtxPopCurrent=0 — while the accumulated err is -22. The context was healthy throughout; a feature extractor's return value shared a variable with the driver's and got reported under the driver's name.

Root cause: flush_context_threaded()'s first loop flushed every TEMPORAL extractor, GPU ones included, while its second loop already skipped VMAF_FEATURE_EXTRACTOR_CUDA deliberately. That asymmetry is the bug.

testdata/bench_all.sh hard-codes --threads 1, so its GPU rows were masked failures for as long as this existed.

Why the obvious fix is wrong (measured, not argued)

Draining the tail before the boundary collect does not only cause a duplicate write. It also emits the last batch-boundary frame without the min() against the following frame that motion2/motion3 are defined by. I built the narrow fix — extend the existing guard to cover the collect as well as the flush — and measured it:

	pooled VMAF	frame 39 integer_motion2_mmxv_18
CPU serial	82.816059	3.724276
CPU --threads 4	82.816059	3.724276
CUDA serial	82.814059	3.724278
CUDA --threads 4, narrow fix	82.823778	4.382255
CUDA --threads 4, this PR	82.814059	3.724278

Every other feature was bit-identical; exactly one frame was wrong. The narrow fix trades a crash for a silently wrong score, which is strictly worse, so it was rejected on evidence.

What this does

flush_context_threaded() no longer touches GPU extractors at all — its first loop now skips CUDA/SYCL flags, matching its second loop — and flush_context_cuda() / flush_context_sycl() own them in both modes, running collect-then-flush in the order the serial path uses. The thread-pool special case in flush_context_cuda() is deleted, not moved. SYCL had the same defect and never had a guard in any form.

flush_context_cuda() also keeps the extractor result and the driver result in separate variables, so "context could not be synchronized" now means that.

Type

• fix — bug fix
• sycl / cuda / simd — backend-specific

Checklist

• Commits follow Conventional Commits.
• make format && make lint is green locally — pre-commit run --files clean on all touched files; clang-tidy -p build core/src/libvmaf.c reports 0 findings, unchanged from master (the added condition pushed flush_context_threaded over the ADR-0141 size budget, so flush_non_temporal_cpu_extractors() was split out rather than suppressed with a NOLINT).
• Unit tests pass: meson test --suite=fast → 139 ok, 4 failed. Those 4 (test_cuda_float_adm_parity, test_cuda_psnr_hvs_parity, test_cuda_ssimulacra2_parity, test_vmaf_cuda_gpumask) fail identically on unmodified origin/master, verified by reverting only libvmaf.c in the same build tree and re-running them.
• If I touched any SIMD/GPU code path — threaded output is bit-identical to serial on CUDA and SYCL at N = 1, 2, 4, 8, compared per-frame across all 48 frames and every metric key, so the worst delta is 0 ULP. GPU-vs-CPU is unchanged by this PR and remains non-bit-exact by design.
• If I touched a feature extractor with SIMD/GPU twins, I updated every twin. — no feature extractor is touched; the change is in the context flush paths.
• If I added a new .c / .cpp / .cu / .h / .hpp, it has the license header. — no new C source; the new file is a shell test.
• If this is a breaking change… — not breaking. A command that used to abort now succeeds; no working invocation changes behaviour.
• If this PR adds an ADR, the row lives in docs/adr/_index_fragments/ and the slug is in _order.txt — number allocated with scripts/adr/next-free.sh --claim; README.md regenerated and --checked.

Bug-status hygiene (ADR-0165)

• docs/state.md — T-GPU-CLI-THREADS-CTX-SYNC-2026-09-06 added to Recently closed with the misreport evidence, both consequences, the rejected narrow fix and its measured wrong score, and the bench_all.sh follow-up.

Netflix golden-data gate (ADR-0024)

• I did not modify any assertAlmostEqual(...) score in the Netflix golden Python tests.

Cross-backend numerical results

feature                     cpu-vs-cuda   cpu-vs-sycl   (serial vs threaded, same backend)
pooled vmaf                  unchanged     unchanged     0 ULP  (82.814059 both, N=1,2,4,8)
integer_motion2_mmxv_18      unchanged     unchanged     0 ULP  (all 48 frames)
all other metric keys        unchanged     unchanged     0 ULP  (all 48 frames)

Deep-dive deliverables (ADR-0108)

• Research digest — no digest needed: trivial. The investigation was instrumentation of one function; its findings are recorded in ADR-1197's Context rather than a separate digest.
• Decision matrix — ADR-1197 ## Alternatives considered weighs five options, including the narrow guard extension that was built and measured before rejection.
• AGENTS.md invariant note — no rebase-sensitive invariants: the two that matter are recorded in docs/rebase-notes.md, which is where this repo keeps core/src/libvmaf.c ones.
• Reproducer / smoke-test command — below.
• CHANGELOG fragment — changelog.d/fixed/adr-1197-gpu-threads-flush.md; CHANGELOG.md regenerated and --checked.
• Rebase note — docs/rebase-notes.md, fix/gpu-threads-ctx-sync — threaded flush leaves GPU extractors alone (2026-09-06).

Reproducer

# Fails on master with exit 234, passes with this change:
meson setup build core -Denable_cuda=true -Denable_sycl=false -Db_lto=false
ninja -C build
meson test -C build test_vmaf_cuda_threads --print-errorlogs

# Or directly — note --threads 1 is enough to trigger it:
./build/tools/vmaf \
  --reference python/test/resource/yuv/src01_hrc00_576x324.yuv \
  --distorted python/test/resource/yuv/src01_hrc01_576x324.yuv \
  --width 576 --height 324 --pixel_format 420 --bitdepth 8 \
  --backend cuda --threads 1

The test asserts the threaded result equals the serial one per frame, not merely that the process exits 0 — checking only the exit code would have passed the wrong-score variant above.

Known follow-ups

• testdata/bench_all.sh GPU numbers need re-taking. It pins --threads 1, so every GPU row it recorded while this bug existed was a failure, not a measurement. Related: fix(testdata): make the Netflix benchmark harness honest and portable (ADR-1192) #1334, which re-ran that harness and surfaced the --threads reproducer this PR fixes.
• Not the FFmpeg-filter nondeterminism. fix(testdata): make the Netflix benchmark harness honest and portable (ADR-1192) #1334 also reports libvmaf_cuda returning a non-modal score in ~10 of 40 runs. That is a distinct defect and is not addressed here; I verified separately that fix(cuda): scope the drain batch to its engine and fence the indexed reads (Netflix/vmaf#1305) #1321 does not fix it either.