Skip to content

fix(cuda): scope the drain batch to its engine and fence the indexed reads (Netflix/vmaf#1305) - #1321

Merged
lusoris merged 1 commit into
masterfrom
fix/cuda-drain-batch-per-state-lifetime
Sep 6, 2026
Merged

lusoris merged 1 commit into
masterfrom
fix/cuda-drain-batch-per-state-lifetime

Conversation

@lusoris

@lusoris lusoris commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Summary

Closes T-UPSTREAM-1305-CUDA-DRAIN-BATCH-THREAD-GLOBAL-2026-09-03, which is two defects of the same shape.

Fork half. The ADR-0242 fence batch in core/src/cuda/drain_batch.c was static _Thread_local and keyed by nothing else, so two VmafContexts sharing an OS thread shared the batch. A context that closed left its CUevents and its bool * drained flags registered, and the next context's flush waited on destroyed events and wrote through freed pointers. The batch now carries the owning VmafCudaState: open() claims it and drops a previous owner's entries, flush() returns 0 without touching CUDA for a foreign owner and clears the entries it consumed, and thread_destroy() wipes entries, the open flag and the owner before the caller frees the engine state.

Upstream half (Netflix/vmaf#1305). vmaf_score_at_index() and vmaf_feature_score_at_index() read the feature collector with no CUDA sync, no drain flush and no thread-pool wait, so a caller draining index N-2 while N was in flight read unwritten slots. They now call a fence_for_read() helper — worker-thread wait, drain flush, pending collect for indices at or below the requested one — but only after the lock-free read reports the slot unwritten, so reads that hit written slots keep their cost and the streaming path keeps its throughput.

Verified on the RTX 4090: CUDA vs CPU on the Netflix src01 pair is unchanged (adm2/adm3/adm_scale0-3/aim/vif_scale0-3 identical, motion2 / motion3 / vmaf 1e-6, the documented GPU tolerance); the Netflix golden gate is 271 passed / 12 skipped / 0 failed; the new unit test pins the ownership rules and needs nvcc but no device.

Type

  • fix — CUDA engine lifetime + public read-API correctness

Checklist

  • Commits follow Conventional Commits.
  • make format && make lint is green locally (pre-commit on every touched file; assertion-density.sh PASS).
  • Unit tests: core/test/test_cuda_drain_batch.c (4 cases, fast suite, no device needed).
  • Docs — no docs needed: no user-visible surface changes; the behaviour change is that a documented API stops returning stale data, and it is recorded in docs/state.md and the CUDA AGENTS.md invariant.
  • SIMD/GPU, twins, new C sources, breaking change, ADR — CUDA engine only; no twin, no new public symbol (vmaf_cuda_drain_batch_pending() is in the private drain_batch.h), no ADR (ADR-0242 already governs the batch).

Bug-status hygiene (ADR-0165)

  • docs/state.md — T-UPSTREAM-1305-CUDA-DRAIN-BATCH-THREAD-GLOBAL-2026-09-03 moved to Recently closed with both halves and the evidence.

Netflix golden-data gate (ADR-0024)

  • I did not modify any assertAlmostEqual(...) score in the Netflix golden Python tests.

Deep-dive deliverables (ADR-0108)

  • Research digest — no digest needed: the state.md row already carries the analysis, and the fix follows the ownership rule ADR-0242 implies.
  • Decision matrix — no alternatives: only-one-way fix. A thread-local batch shared by two engines is wrong under any design; the alternatives (a per-context batch allocation, or dropping the batch) either change ADR-0242's shape or lose its optimisation.
  • AGENTS.md invariant note — core/src/cuda/AGENTS.md § Lifecycle invariants: the drain batch belongs to one engine at a time; never restore the owner-less open(void) signature.
  • Reproducer / smoke-test command — below.
  • CHANGELOG fragment — changelog.d/fixed/cuda-drain-batch-per-state-lifetime.md.
  • Rebase note — docs/rebase-notes.md entry (core/src/libvmaf.c is upstream-mirror; the two hunks there are fork-only).

Reproducer

meson setup core/build core -Denable_cuda=true -Denable_sycl=false -Db_lto_threads=4 && ninja -C core/build
core/build/test/test_cuda_drain_batch                      # 4 tests run, 4 passed
Y=python/test/resource/yuv
core/build/tools/vmaf -r $Y/src01_hrc00_576x324.yuv -d $Y/src01_hrc01_576x324.yuv -w 576 -h 324 -p 420 -b 8 \
  --backend cuda --model version=vmaf_v0.6.1 --json -o cuda.json
core/build/tools/vmaf -r $Y/src01_hrc00_576x324.yuv -d $Y/src01_hrc01_576x324.yuv -w 576 -h 324 -p 420 -b 8 \
  --no_cuda --model version=vmaf_v0.6.1 --json -o cpu.json   # adm/vif identical, motion/vmaf within 1e-6
make VENV=/path/to/.venv test-netflix-golden               # 271 passed, 12 skipped

🤖 Generated with Claude Code

@lusoris lusoris added this to the 1.0.0 — First release milestone Sep 5, 2026
@lusoris
lusoris force-pushed the fix/cuda-drain-batch-per-state-lifetime branch 2 times, most recently from 71c011d to 50a368b Compare September 5, 2026 23:12
…reads (Netflix/vmaf#1305)

Two defects of the same shape, one row (docs/state.md
T-UPSTREAM-1305-CUDA-DRAIN-BATCH-THREAD-GLOBAL-2026-09-03).

Fork half: the ADR-0242 fence batch in core/src/cuda/drain_batch.c was
`static _Thread_local` and keyed by nothing else, so two VmafContexts sharing an
OS thread shared the batch. A context that closed left its CUevents and its
`bool *` drained flags registered, and the next context's flush waited on
destroyed events and wrote through freed pointers. The batch now carries the
owning VmafCudaState: open() claims it and drops a previous owner's entries,
flush() returns 0 without touching CUDA for a foreign owner and clears the
entries it consumed, and thread_destroy() wipes entries, the open flag and the
owner before the caller frees the engine state.

Upstream half: vmaf_score_at_index() and vmaf_feature_score_at_index() read the
feature collector with no CUDA sync, no drain flush and no thread-pool wait, so
a caller draining index N-2 while N was in flight read unwritten slots. They now
call fence_for_read() — worker-thread wait, drain flush, pending collect for
indices <= the requested one — but only after the lock-free read reports the
slot unwritten, so reads that hit written slots keep their cost.

Verified on the RTX 4090: CUDA vs CPU on the Netflix src01 pair is unchanged
(adm/vif/aim identical, motion2/motion3/vmaf 1e-6, the documented GPU
tolerance); core/test/test_cuda_drain_batch.c (4 cases, needs nvcc but no
device) pins the ownership rules.
@lusoris
lusoris force-pushed the fix/cuda-drain-batch-per-state-lifetime branch from 50a368b to d5ba773 Compare September 6, 2026 00:01
@lusoris
lusoris marked this pull request as ready for review September 6, 2026 00:16
@lusoris
lusoris merged commit e3fa7e6 into master Sep 6, 2026
116 of 117 checks passed
@lusoris
lusoris deleted the fix/cuda-drain-batch-per-state-lifetime branch September 6, 2026 00:40
@lusoris lusoris added the type:bug Something isn't working label Sep 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type:bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant