fix(testdata): make the Netflix benchmark harness honest and portable (ADR-1192)

[lusoris]

Summary

Re-ran the Netflix benchmark suite on current master (cd52f2670) for epic #1245 items 1 and 5, and made the harness honest and portable so the next person can reproduce it. The scores do not match the recorded snapshot on any backend, so per the epic's own gate no fresh throughput baseline is recorded and testdata/netflix_benchmark_results.json is deliberately left untouched (ADR-1192). The run also reproduced two GPU defects — both confirmed pre-existing by rebuilding the commit before today's GPU merges.

Type

• feat — new feature
• fix — bug fix
• perf — performance improvement
• refactor — no behavior change
• docs — documentation only
• test — test-only
• build / ci — tooling / infra
• port — cherry-pick from upstream Netflix/vmaf
• sycl / cuda / simd — backend-specific

What was measured

Host ryzen-4090-arc: AMD Ryzen 9 9950X3D (32 threads), 60 GiB, Linux 7.2.3-1-cachyos, RTX 4090 (driver 610.57.04), Arc A380. All runs inside vmaf-dev-mcp (Ubuntu 26.04, oneAPI DPC++ 2026.1.1, CUDA 13.3.73, FFmpeg n9.0.1-17-gfde3691) against an out-of-source libvmaf built --buildtype=release -Db_lto=false -Dc_args=-march=native and injected via LD_LIBRARY_PATH. The workstation was not idle — 1-minute load average 7–19 throughout (a container image build plus other agent sessions); the timing sweep specifically ran at load 9.6–9.8.

Scores versus testdata/netflix_benchmark_results.json (CUDA at its modal value)

Fixture	Backend	Recorded	2026-09-06	Pooled delta	Max per-frame
src01_576x324	cpu	76.667828	76.667831	+2.83e-06	1.70e-05
src01_576x324	cuda	76.668903	76.667830	-1.07e-03	2.85e-03
src01_576x324	sycl	76.669148	76.667746	-1.40e-03	1.54e-02
checker_1080p_mild	cpu	35.068672	35.068671	-6.67e-07	1.30e-05
checker_1080p_mild	cuda	35.068669	35.068667	-2.33e-06	5.00e-06
checker_1080p_mild	sycl	35.068664	35.068628	-3.60e-05	3.90e-05
checker_1080p_heavy	cpu / cuda / sycl	7.985899	7.985899	0	≤1.00e-06

None of this is today's merges. A rebuild of 5a080300e (immediately before #1307, #1312, #1324) with identical flags gives CUDA 76.667830 and SYCL 76.667745 on the 576x324 pair — the same drift versus a snapshot last written by PR #309 on 2026-05-02. What the three merges did change: CPU per-frame values moved by up to ~8e-6 (pooled unchanged at six decimals) and the SYCL frames[].metrics key set shrank 35 → 24. The direction of the four-month drift is CUDA and SYCL converging towards CPU (recorded CUDA sat 1.07e-3 above CPU; it now tracks CPU to 2e-5).

Throughput — observation only, not a recorded baseline

Median of 5, whole-process wall time, load 9.6–9.8. Published in the new docs page; not written into docs/benchmarks.md, because a timing table next to a CUDA score that is wrong a quarter of the time would be misleading.

Fixture	cpu	cuda	sycl
src01 576x324, 48f	98 ms (95–106), 489.8 fps	185 ms (180–200), 259.5 fps	250 ms (237–326), 192.0 fps
checkerboard 1080p, 3f	82 ms (74–103), 36.6 fps	171 ms (168–180), 17.5 fps	160 ms (152–204), 18.8 fps

Two GPU defects reproduced (both pre-existing)

1. vmaf --threads N aborts on every GPU backend — T-GPU-CLI-THREADS-CTX-SYNC-2026-09-06. Exit 234, context could not be synchronized, no output. Drop --threads and both backends succeed and are bit-stable 10/10 (CUDA 76.667830, SYCL 76.667746). bench_all.sh hard-codes --threads 1, so its GPU rows were masked failures.
2. libvmaf_cuda FFmpeg filter is non-deterministic — T-CUDA-FFMPEG-FILTER-NONDETERMINISM-2026-09-06. 10 of 40 runs on cd52f2670 and 8 of 40 on 5a080300e returned a non-modal pooled score; bad runs corrupt one or two individual frames (frame 1 = 0.0 vs CPU 82.639803). CPU 10/10 and SYCL 10/10 through the same FFmpeg are bit-stable, and CUDA through the CLI with no thread pool is 10/10 stable. 20 % vs 25 % over n=40 each is inside binomial noise, so the merges neither caused nor measurably worsened it.

Both match the statically-derived T-UPSTREAM-1305-CUDA-DRAIN-BATCH-THREAD-GLOBAL-2026-09-03 row; these runs are the empirical reproducer that row said the fix PR would need. The mechanism is not proven here — only the symptom, the thread-pool dependency and the pre-existence.

Checklist

• Commits follow Conventional Commits (the commit-msg hook enforces this).
• make format && make lint is green locally — pre-commit run --files on all 14 touched files passes (shellcheck, shfmt, markdownlint, black, ruff, semgrep, ADR + venv + model-single-source gates).
• Unit tests pass: meson test -C build. — no C/C++ source touched; the changed files are two testdata/ harness scripts and documentation.
• If I touched any SIMD/GPU code path, I ran /cross-backend-diff and the worst ULP is ≤ 2. — no GPU source touched; the cross-backend numbers this PR reports are measurement output, not a code change.
• If I touched a feature extractor with SIMD/GPU twins, I either updated every twin or listed the gap under "Known follow-ups" below. — no extractor touched.
• If I added a new .c / .cpp / .cu / .h / .hpp, it has the appropriate license header. — none added.
• If this is a breaking change, the commit message uses ! or BREAKING CHANGE:. — not breaking.
• If this PR adds an ADR, the ADR row lives in docs/adr/_index_fragments/<NNNN-slug>.md and the slug is appended to docs/adr/_index_fragments/_order.txt.

Bug-status hygiene (ADR-0165)

• docs/state.md updated in this PR with a row in the appropriate section — two new Open-bug rows plus a dated update note.

Netflix golden-data gate (ADR-0024)

• I did not modify any assertAlmostEqual(...) score in the Netflix golden Python tests.
• No snapshot regenerated either — testdata/netflix_benchmark_results.json is untouched by design (ADR-1192).

Cross-backend numerical results

src01_576x324, vmaf_v0.6.1, FFmpeg filter path, cd52f2670
cpu   76.667831   (10/10 runs identical)
cuda  76.667830   max |cpu-cuda| per frame 2.0e-05   (30/40 runs; 10/40 return a wrong pooled value)
sycl  76.667746   max |cpu-sycl| per frame 1.5e-02   (10/10 runs identical)

Performance (if perf or feat)

See "Throughput" above. No performance change is claimed or intended by this PR.

Deep-dive deliverables (ADR-0108)

• Research digest — docs/research/2026-09-06-netflix-benchmark-rerun.md.
• Decision matrix — docs/adr/1192-netflix-bench-snapshot-drift-not-regenerated.md ## Alternatives considered (regenerate now / delete the CUDA rows / loosen tolerance / keep and record).
• AGENTS.md invariant note — core/AGENTS.md §"Backend-engagement foot-guns" gains the --threads-aborts-GPU and never-discard-stderr invariants plus the updated key counts.
• Reproducer / smoke-test command — below.
• CHANGELOG fragment — changelog.d/fixed/netflix-bench-harness-honesty.md, rendered via scripts/release/concat-changelog-fragments.sh --write.
• Rebase note — docs/rebase-notes.md RN-2026-09-06.

Reproducer

# Harness now runs all three backends and reports real failures.
docker exec vmaf-dev-mcp bash -lc '
  set +u; source /opt/intel/oneapi/setvars.sh >/dev/null 2>&1; set -u
  export LD_LIBRARY_PATH=/tmp/bench-build/src:$LD_LIBRARY_PATH
  export VMAF_YUVDIR=/workspace/python/test/resource/yuv
  export VMAF_SYCL_RENDER_NODE=/dev/dri/renderD129
  mkdir -p /tmp/benchrun && cp /workspace/testdata/benchmark_netflix.py /tmp/benchrun/
  cd /tmp/benchrun && python3 benchmark_netflix.py'

# The masked-failure fix: this used to print "SKIP (… backend likely unavailable)".
VMAF_ROOT=/workspace VMAF_BIN=/tmp/bench-build/tools/vmaf \
  VMAF_BENCH_OUTDIR=/tmp/benchall bash testdata/bench_all.sh
#   t1_cuda ... FAIL (vmaf exited 234: problem flushing context; see /tmp/benchall/t1_cuda.err)

Known follow-ups

• Regenerating testdata/netflix_benchmark_results.json is blocked on T-GPU-CLI-THREADS-CTX-SYNC-2026-09-06 and T-CUDA-FFMPEG-FILTER-NONDETERMINISM-2026-09-06 closing (ADR-1192). The regenerating PR must cite that ADR.
• docs/benchmarks.md still carries its 2026-05 throughput table and references a make bench target that does not exist in the Makefile. Left alone here to keep this PR scoped.
• The /run-netflix-bench skill points at testdata/compare_combined.py, which compares scores_cpu_*.json against scores_sycl_a380_*.json and never reads the Netflix snapshot. The new docs page records the mismatch; fixing the skill is a separate change.

[lusoris]

The --threads half of this is now root-caused and fixed in #1343, so the hypothesis recorded here can be replaced with a finding — worth doing before ADR-1192 lands, since it currently carries an attribution I have since disproved.

It is not T-UPSTREAM-1305-CUDA-DRAIN-BATCH-THREAD-GLOBAL. I built #1321 (the drain-batch fix) with CUDA and ran the reproducer against it: --threads 1, 4 and 8 all still exit 234. That PR does not fix this.

The real mechanism. Instrumenting flush_context_cuda() shows all four driver calls returning success — cuCtxPushCurrent=0 cuStreamSynchronize=0 cuCtxSynchronize=0 cuCtxPopCurrent=0 — while the accumulated err is -22. The context was healthy the whole time; a feature extractor's return value shared a variable with the driver's result and was reported under the driver's name. flush_context_threaded()'s first loop flushed every TEMPORAL extractor including GPU ones, though its second loop already skipped VMAF_FEATURE_EXTRACTOR_CUDA deliberately. That ran a temporal GPU extractor's tail-batch drain before the pending boundary collect, so the later collect was a duplicate write → -EINVAL.

A detail that matters for your throughput table. The duplicate write is not the only consequence. Draining the tail early also emits the last batch-boundary frame without the min() against the following frame that motion2/motion3 are defined by. I built the narrow fix (skip the redundant collect) and measured it: pooled VMAF 82.823778 against serial 82.814059, frame 39 reading 4.382255 instead of 3.724278. So a fix that only silenced the crash would have produced usable-looking GPU throughput rows carrying a wrong score. #1343 instead gives output bit-identical to serial at N = 1, 2, 4, 8, per-frame across all 48 frames.

Two things from your findings I would keep exactly as they are:

• Your bench_all.sh observation is the important one and fix(gpu): let the backend flush own GPU extractors under --threads (ADR-1197) #1343 cites it: it hard-codes --threads 1, so its GPU rows were masked failures for as long as this existed. Its recorded GPU numbers need re-taking once fix(gpu): let the backend flush own GPU extractors under --threads (ADR-1197) #1343 merges.
• The FFmpeg-filter nondeterminism (T-CUDA-FFMPEG-FILTER-NONDETERMINISM-2026-09-06) is a separate, still-open defect. fix(gpu): let the backend flush own GPU extractors under --threads (ADR-1197) #1343 does not address it, and neither does fix(cuda): scope the drain batch to its engine and fence the indexed reads (Netflix/vmaf#1305) #1321 — I checked. Your 10/40 and 8/40 measurement stands on its own.

Also confirming your refusal to regenerate testdata/netflix_benchmark_results.json on a score mismatch was the right call, and the four-month drift you bisected to before 5a080300e is independent of today's merges.

[lusoris]

Follow-up measurement on T-CUDA-FFMPEG-FILTER-NONDETERMINISM-2026-09-06, since #1343, #1321 and #1325 have all merged since your run. The bug is still there — the row should stay Open.

Current master ee43938f1, libvmaf_cuda through ffmpeg, 576x324 48f pair, libvmaf built CC=icx CXX=icpx --buildtype=release -Db_lto=false and injected via LD_LIBRARY_PATH:

n=80:  78 x 76.667830 (modal)
        2 x 74.946168

2/80 = 2.5%, against your 10/40 = 25% on cd52f2670. I would not read the drop as an improvement without more data — my n=80 and your n=40 differ in compiler and flags, and the wrong value 74.946168 is one your runs saw too.

A correction to my own work, so it is not repeated. I ran what looked like a clean A/B — 0/80 wrong with #1343's tree versus 11/80 on a "master" tree, p≈0.0006 — and it was confounded: the control worktree had drifted to master+#1325+#1321 while my branch sat at merge-base cd52f2670, so the two trees differed by two extra PRs, not by my one diff. The comparison supports nothing about #1343 and I am discarding it.

It could not have supported it anyway, which is the useful part: the libvmaf_cuda filter defaults n_threads to 0, and core/src/libvmaf.c:249 returns early without creating a thread pool when n_threads == 0. flush_context_threaded() is therefore never entered in these runs, so #1343's change is not on this code path at all. Verified directly rather than assumed — with n_threads=4 passed explicitly, a master build prints context could not be synchronized and a #1343 build does not; with the default, neither errors.

So: #1343 fixes the filter's explicit n_threads=N failure, and has nothing to do with the default-path nondeterminism you found. That one still needs its own root cause, and the corrupted-single-frame signature you documented (frame 1 = 0.0 against CPU 82.639803) is still the best lead.