Skip to content

Fix "context could not be synchronized" abort with CUDA + --threads - #1538

Open
dsummer wants to merge 1 commit into
Netflix:masterfrom
codecmarket:fix/cuda-multithread-context-sync
Open

dsummer wants to merge 1 commit into
Netflix:masterfrom
codecmarket:fix/cuda-multithread-context-sync

Conversation

@dsummer

@dsummer dsummer commented May 29, 2026 •

Copy link
Copy Markdown

Summary

Running vmaf --gpumask 0 --threads N aborts at flush with
context could not be synchronized (exit 234), preceded by a flood of
feature ... cannot be overwritten at index N warnings.

Cause

The read dispatch in vmaf_read_pictures carries the note "multithreading for
GPU does not yield performance benefits / disabled for now"
, but GPU feature
threading is not actually disabled. When a thread pool is present,
vmaf_read_pictures returns early via threaded_read_pictures_batch, which:

  • bypasses the HAVE_CUDA device-picture cleanup below it — the
    cuEventRecord(finished) and the vmaf_picture_unref of
    ref_device/dist_device; and
  • flushes feature extractors from worker threads that lack the bound CUDA
    context.

The CUDA context is therefore left unsynchronized and flush_context fails.

Fix

Tear the thread pool down in vmaf_cuda_import_state so CUDA runs take the
single-threaded path, matching the existing comment's intent. GPU feature
threading yields no throughput benefit (the GPU pipeline, not host
feature-extraction parallelism, is the bound).

Validation

RTX 5090, CUDA 13.2, vmaf_v0.6.1, 1080p y4m:

invocation before after
--gpumask 0 (single-thread) OK OK
--gpumask 0 --threads 4 exit 234 (context could not be synchronized) OK, score identical to single-thread

Re-verified on the v3.2.0 release (master ac9467ff)

The bug still reproduces on current master, and this fix still applies
cleanly — vmaf_cuda_import_state is unchanged since this PR was opened, so the
patch cherry-picks without conflict. Built pure upstream master with CUDA
(-Denable_cuda=true -Denable_float=true -Denable_nvcc=true) on an RTX 5090 /
CUDA 13.3, 854p y4m:

invocation before fix after fix
--gpumask 0 --threads 0 OK (vmaf 85.194703) OK
--gpumask 0 --threads 4 exit 234 OK, vmaf 85.194703 (bit-identical to single-thread)
--threads 4 (CPU, no CUDA import) OK OK (no regression)

Running `vmaf --gpumask 0 --threads N` aborts at flush with
"context could not be synchronized" (exit 234), preceded by a flood of
"feature ... cannot be overwritten at index N" warnings.

The read dispatch in vmaf_read_pictures already notes that "multithreading
for GPU does not yield performance benefits / disabled for now", but it is
not actually disabled: when a thread pool is present the function returns
early via threaded_read_pictures_batch, bypassing the HAVE_CUDA device-
picture cleanup below it (the cuEventRecord(finished) and the unref of
ref_device/dist_device), and feature extractors flush from worker threads
that lack the bound CUDA context.

Tear the thread pool down in vmaf_cuda_import_state so CUDA runs take the
single-threaded path, matching the existing comment's intent. GPU feature
threading yields no throughput benefit.

Reproduced on master with an RTX 5090 / CUDA 13.2: before, exit 234; after,
completes with scores identical to the single-threaded path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@dsummer

dsummer commented Jun 21, 2026

Copy link
Copy Markdown
Author

Re-verified against the v3.2.0 release (master @ ac9467ff) on an RTX 5090 / CUDA 13.3.

  • The crash still reproduces on a clean upstream CUDA build: vmaf --gpumask 0 --threads 4 → exit 234 (context could not be synchronized), with the ... cannot be overwritten at index N warnings.
  • This PR's patch cherry-picks cleanly onto current master (the vmaf_cuda_import_state body is unchanged), and after applying it: --gpumask 0 --threads 4 → exit 0 with a score bit-identical to the single-threaded path (85.194703), and the CPU --threads path is unaffected.

The branch is based on e4b93c6e but remains mergeable against current master. Happy to rebase or squash if that's preferred.

@lusoris

lusoris commented Oct 1, 2026

Copy link
Copy Markdown

Tested on master 8e7a1ac4e + this PR (merges without conflicts), RTX 4090, CUDA 13.4.92, driver 615.71.09, gcc 16.2.1, release build with -Denable_cuda=true.

The abort reproduces on master and is gone with the PR. vmaf --gpumask 0 --threads N on src01 576x324, 48 frames, vmaf_v0.6.1: master exits 234 (context could not be synchronized) for --threads 2 and 4, exit 0 for --threads 0. With the PR all three exit 0, and every metric of every frame is identical to the master --threads 0 run (max abs difference 0; the CLI prints 6 digits). meson test: 25 ok, 1 fail (test_cuda_pic_preallocation, SIGSEGV), same as master.

The cause on this machine is not the one in the description. The line before the error is a single libvmaf WARNING feature "VMAF_integer_feature_motion2_score" cannot be overwritten at index 47: motion_cuda carries both TEMPORAL and CUDA, so flush_context_threaded() flushes it in the temporal loop and the HAVE_CUDA block flushes it again, the second append of the last score returns -EINVAL, and because flush_context() accumulates every flush result in err, the check at libvmaf.c:708 prints it as context could not be synchronized. A guard that skips CUDA extractors in the temporal loop (the first commit of #1583; #1553 has the same hunk plus a test) is enough on its own: master + #1553 exits 0 for --threads 0 and 4, with scores identical to the master --threads 0 run, and the thread pool stays.

That is the cost of the approach here. Destroying the pool in vmaf_cuda_import_state() also removes the threads for CPU extractors that run next to CUDA ones. Wall time, same pair, vmaf_v0.6.1 plus --feature ciede --feature float_ssim --feature psnr_hvs, --gpumask 0, three runs each:

build threads wall time
master 0 3.23, 3.26, 3.38 s
this PR 8 3.41, 3.40, 3.48 s
#1583 (flush guard, pool kept) 8 0.64, 0.64, 0.62 s

Overlap: it merges textually with #1583, #1553 and #1613, but #1613 and #1583 keep CPU threading on CUDA runs, which this PR turns off.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants