feat(api): split libvmaf into libvmafx.so.1 and a compat libvmaf.so.3 on the VMAFx API (RC4 WP6)

[lusoris]

Summary

RC4 work package 6: the library is split into libvmafx.so.1 (engine + VMAFx API, vmafx_* exports only) and a thin libvmaf.so.3 that implements the libvmaf API on the exported vmafx_* functions alone, so a libvmaf function the new API cannot express fails the link (ADR-1852 decision D3). WP5 #2288 is on master; this PR's diff is against master. Implementation decisions are in ADR-2094 (Accepted by maintainer popup on 2026-10-06: "Forced header (Recommended)", cited in its References; it implements ADR-1852 D3 and D7 and design section 2.11).

What changes:

• 107 of 107 libvmaf functions are [[compat]] entries in core/api/vmafx.toml: 14 generated shims, 5 generated glue, 63 hand-written (core/src/compat/libvmaf/, each declaring the vmafx_* functions it calls), 25 declared engine exceptions (the 5 CUDA and 20 SYCL functions keep the engine's definition until their WP3 lane lands; CUDA device frames are feat(api): import CUDA device frames with event fences and GL textures (RC4 WP3, ADR-2023) #2277). The 5 HIP and 9 Metal functions are compat functions in builds without those backends and engine exceptions in builds with them, until their WP3 lanes. Every exception names what ends it (until) and is exported from libvmafx.so.1 in a version node VMAF_LEGACY_<BACKEND> (core/src/vmafx_legacy_<backend>.map).
• The library split. libvmafx links the engine with hide_unlisted = true (local: *;); libvmaf.so.3 is a separate target linked with -Wl,--no-undefined against libvmafx.so.1. Every engine translation unit compiles with the generated core/src/vmafx/engine_names_gen.h, which names the engine's own libvmaf bodies vmaf_engine_<stem>, so upstream syncs port into the engine sources unchanged. libvmafx.pc and libvmaf.pc (Requires: libvmafx) are generated; pkg-config --libs libvmaf gives -lvmaf -lvmafx.
• VMAFx API 0.1.6 (ADR-1897 patch bump, node VMAFX_0.1): the libvmaf bridge for pictures, models and model sets, context-owned preallocated frames, perceptual side data, the context backend, frame converters, the tiny-AI functions (vmafx/dnn.h), the embedded MCP server (vmafx/mcp.h) and vmafx_backend_name().
• Deprecation (D7). Every libvmaf declaration carries VMAF_DEPRECATED("use <successor>"), a warning only with -DVMAF_ENABLE_DEPRECATION_WARNINGS (opt-in in 1.0, default in 1.1, removal in 2.0).
• Upstream consumers. scripts/ci/upstream-ffmpeg-compat.sh (unpatched FFmpeg FFMPEG_TAG, libvmaf filter, --cuda for libvmaf_cuda) and scripts/ci/upstream-gstreamer-compat.sh (unpatched gst-plugins-bad vmaf element, pinned GST_PLUGINS_BAD_VERSION with a Renovate git-tags manager, CI conformance: upstream GStreamer vmaf element on the libvmaf.so.3 compatibility layer #2237) build against an installed library and compare scores with the CLI and a reference library as exact text; .github/workflows/upstream-consumers.yml runs both.
• Repository consumers follow the split: container images copy libvmafx.so*, Go builds link -lvmaf -lvmafx, the MCP CI lane runs the compat checks.

--abi-check --against-ref origin/rc4/api-generation-prototype: definition is an append-only successor of origin/rc4/api-generation-prototype (347 additions); against origin/rc4/api-wp5-provenance: 84 additions. No break of a prototype entry, so no new minor.

no ffmpeg-patches update needed: the libvmaf headers gain only VMAF_DEPRECATED markers, empty unless the consumer defines VMAF_ENABLE_DEPRECATION_WARNINGS; unpatched upstream FFmpeg n9.0.2 builds against the split library and scores identically (below).

Landing (Q-083)

Lands bottom-up per Q-083: v1.0.0-rc.3 is tagged, so RC4 lands through the merge train, one API PR at a time. Its base #2288 is on master; this PR was rebased onto master ae8ccbb23 (the base's commits dropped), retargeted to master, and #2277 (WP3 CUDA) follows once it lands. The API docs keep marking the VMAFx API as a preview (docs/api/vmafx/index.md, ABI 0.x) until the rc.4 cut. ADR-2094 is Accepted.

ABI check against master: python3 scripts/codegen/vmafx-api.py --abi-check --against-ref origin/master -> definition is an append-only successor of origin/master (86 additions). Additions only, node VMAFX_0.1, abi_version 0.1.6 (patch bump over master's).

Rebase: generated files take master's side and are regenerated once at the tip (vmafx-api.py --write, the AGENTS indexes); under render at landing (ADR-2197) this PR carries no rendered file (CHANGELOG.md, the ADR index, tag and title pages, docs/rebase-notes.md): its rebase note is a fragment under docs/rebase-notes.d/, and the citation map is derived from the tree (ADR-2200); docs/state.md by scripts/dev/resolve-state-md-conflict.py.
Two contracts master gained since this draft needed changes here, each its own commit: the e2e runtime contract test pinned the libvmaf-only SONAME find of the image builders, so it now pins both chains and both .pc files the split copies (scripts/ci/test_e2e_runtime_contract.py); and master's CI tiers (ADR-2169) route every pull-request workflow through ci-tier.yml, so upstream-consumers.yml gains the tier job and runs its 45-minute FFmpeg and GStreamer build at the full tier (master push, dispatch, fork pull request, ci: full), like the FFmpeg lanes; docs/development/upstream-consumers.md and the workflow table of docs/development/ci.md say so. The compat library also carries the two libvmaf functions newer than this draft (vmaf_set_sample_range_check_enabled(), vmaf_set_input_colorimetry(), ADR-2094 plan).

A CUDA build of the stack failed test_compat_conformance: after vmaf_read_pictures() the engine left the caller's VmafPicture structs set (its CUDA path releases host translations that are struct copies of them), while the compat library clears them (scoring scenario consumed 0 0 against consumed 1 1). vmaf_engine_read_pictures() now wraps read_pictures_owned() and clears both caller structs once the context owns the pictures, in every build; the GPU fallback source contract reads the helper's body (state row T-ENGINE-CUDA-READ-LEAVES-CALLER-PICTURES-2026-10-07, rebase note). On the previous base a CUDA build of the stack gave 0 warnings and --suite=fast 482 OK, 0 fail. tidy: cpu core/src/libvmaf.c 0 findings.

Two more fixes for the stack and master: core/src/vmafx_legacy_sycl.map named vmaf_sycl_import_d3d11_surface, which only _WIN32 builds define, so every Linux SYCL link failed (--no-undefined-version). [[compat]] gains platform = "windows": the ELF legacy map leaves the name out, its row in core/src/libvmaf_symbols.txt ends in @windows, and check_exported_symbols.py expects it only on Windows. Two generator tests fail on the previous head. And the Rust tests master gained (test_rust_twin_registry from #2086, test_predict_rust_ops and test_rust_predict from #2085, test_rust_cambi_kernels from #2090) asked for libvmaf.get_static_lib(), which the split removes, so meson setup -Denable_rust_features=true failed: they link vmaf_test_link now (grep -c get_static_lib core/test/meson.build is 0); a Rust-enabled configure succeeds and the four tests pass. The rebase note is the fragment docs/rebase-notes.d/vmafx-libvmaf-compat-split.md (ADR-2197). The pre-push assertion-density gate refused vmaf_mcp_start_sse() in core/src/compat/libvmaf/mcp.c (20 lines, no assertion): it now asserts that a successful start had a configuration and that the bound port fits the 16-bit field (clang-tidy cpu: 0 findings).

Local gate on the rebased head (CPU, -Db_lto=false, -j4, warnings as errors): build 0 warnings; --suite=fast 410 OK, 0 fail (test_gpu_picture_pool_uaf on its own with MALLOC_PERTURB_=0, as #2547 sets it: OK); codegen tests 132 passed; affected suites: tooling 2479 passed, 6 skipped, 51 shell suites ok; go vet ./... clean and go test ./... 68 packages ok against this build (CGO_LDFLAGS=-L build-cpu/src -lvmaf -lvmafx -lm, as go-ci.yml now links); make test-netflix-golden GOLDEN_NINJA_JOBS=4 280 passed, 3 skipped; preflight.sh --stage msvcism pass; Rust-enabled build (-Denable_rust_features=true) 0 warnings, --suite rust 5 OK; clang-tidy cpu lane on the 30 translation units this PR touches: 0 findings. CUDA build of the API stack through #2290 on master 4994380 (-Denable_cuda=true, warnings as errors): 0 warnings, --suite=fast 489 OK, 0 fail. New compiler-specific constructs are guarded (VMAF_DEPRECATED has an MSVC __declspec branch and an empty fallback; the trace format attribute and the C23 va_start workaround are GCC/clang-only with plain fallbacks). The merge train builds CPU and CUDA and runs its own gates (deliverables, state-md, silent revert, praetorctl audit).

Type

• feat — new feature

Checklist

• Commits follow Conventional Commits (the commit-msg hook enforces this).
• make format && make lint is green locally — the commit hooks pass.
• Unit tests pass: python3 scripts/ci/run_meson_test.py -- -C build --suite=fast --num-processes 4 -> 377 OK, 2 expected fail (the planted conformance defects), 0 fail, 1 skipped (test_vmafx_api_abi_append_only: the merge base with origin/master has no definition yet) on a CPU build (-Denable_dnn=disabled -Db_lto=false) at f324808b4, three times for the conformance trio run in parallel; the WP6 tests also pass in a DNN build (ONNX Runtime) and an MCP build (-Denable_mcp=true, every transport); scripts/codegen/tests 110 OK; scripts/ci/tests/test_upstream_consumer_scores.py 10 passed; make docs-figures, make docs-fragments-check, the public-API Doxygen build (WARN_AS_ERROR, output identical to before the markers) clean.
• If I touched any SIMD/GPU code path, I ran /cross-backend-diff and the worst ULP is ≤ 2. — not applicable: no SIMD or kernel code touched; GPU host functions stay the engine's (declared exceptions).
• If I touched a feature extractor with SIMD/GPU twins, I either updated every twin or listed the gap under "Known follow-ups" below. — not applicable: no extractor touched.
• If I added a new .c / .cpp / .cu / .h / .hpp, it has the appropriate license header (EUPL-1.2, fork-authored).
• If this is a breaking change, the commit message uses ! or BREAKING CHANGE: and the migration path is documented below. — not a breaking change: the libvmaf ABI is kept (same SONAME, same symbols, a binary linked against the previous libvmaf.so.3 runs unchanged); the definition is append-only.
• If this PR adds an ADR, the ADR row lives in docs/adr/_index_fragments/<NNNN-slug>.md and the slug is appended to docs/adr/_index_fragments/_order.txt — ADR-2094, number claimed with scripts/adr/next-free.sh --claim, indexes regenerated.

tidy: cpu core/src/compat/libvmaf/{context,dnn,libvmaf_gen,model,picture,score,status_errno_gen,backend_absent_hip,backend_absent_metal}.c, core/src/vmafx/{bridge,context_frames,convert,tiny_model,mcp_server,frame_pool,model,context}.c, core/src/{libvmaf,model}.c, core/src/{dict,output}.cpp, core/src/feature/psnr.c, core/tools/vmaf.cpp, core/test/{test_compat_conformance,test_compat_conformance_api,test_compat_conformance_scoring,compat_conformance_table_gen,test_vmafx_abi_layout}.c (and through them the forced engine_names_gen.h, the conformance header and the libvmaf headers with their markers): 0 findings in WP6 files, 0 uncited NOLINT (dev container, clang-tidy 22.1.8, scripts/dev/tidy-lane.sh --only ... cpu). A first run found 21 (two pointer casts through void * between the libvmaf and VMAFx tensor records, now member-wise copies; an analyzer NULL path in vmafx_frame_convert(); the C23 va_start the VAList analyzer does not model; getenv in a test; integer-to-pointer state sentinels; out-of-range enum values the tests pass on purpose, now under NOLINTNEXTLINE(...EnumCastOutOfRange) -- ADR-1080 as core/test/dnn/test_tensor_io.c does; NULL sentinels in the generated header); all fixed. The run also reports the 7 core/include/libvmaf/picture.h findings WP5 and WP8 recorded (pre-existing on this base, from the C++ TUs; origin/master carries their ADR-1470 NOLINT blocks). GPU lanes not measured: no GPU translation unit touched; the forced engine-name header holds #defines only.

scripts/dev/preflight.sh --stage msvcism: pass. praetorctl audit --offline: governance gates passed, HISS scan clean for the touched files (the first commit attempt found 7 HISS-07 shell findings in the new consumer scripts: strict mode in the sourced library and six || true; all fixed).

Bug-status hygiene (ADR-0165)

• docs/state.md updated — new open row T-VMAFX-COMPAT-BACKEND-EXCEPTIONS-2026-10-06: the CUDA / SYCL (and HIP / Metal when built) libvmaf functions remain engine exceptions until their WP3 lanes; the upstream FFmpeg libvmaf_cuda leg is written but unrun (no device on this host); release artifacts stage libvmaf.so* without libvmafx.so* until WP12.

Netflix golden-data gate (ADR-0024)

• I did not modify any assertAlmostEqual(...) score in the Netflix golden Python tests.
• If I believe a golden value must change, I have explained why below — not applicable.

Golden gate through the compat library (core/build-golden built with gcc, GOLDEN_NINJA_JOBS=4, make VENV=<repo>/.venv test-netflix-golden; the CLI the harness runs links libvmaf.so.3 on libvmafx.so.1): 280 passed, 3 skipped at 8a18b9628 and again at f324808b4, the same as WP5.

Cross-backend numerical results

Not applicable: no extractor or kernel changed. Scores through the compat library equal the old libvmaf bodies bit for bit (test_compat_conformance, every score as %a) and the CLI's (FFmpeg and GStreamer below).

Performance (if perf or feat)

No scoring path changed. A libvmaf call costs one extra call through the exported vmafx_* function; vmaf_read_pictures() wraps the pictures as frames by reference (no copy). Generic tiny-AI runs copy up to four tensor records of each kind on the stack (more in the heap), as the engine already does.

Deep-dive deliverables (ADR-0108)

• Research digest — no digest needed: implements Research-2158 section 2.11 and ADR-1852 D3 / D7; the implementation choices (forced engine-name header, engine exceptions, engine pool for preallocation, picture adoption, two-table conformance) are in ADR-2094.
• Decision matrix — docs/adr/2094-libvmaf-compat-library-split.md ## Alternatives considered (renaming every body in the source, partial linking of the static engine, GPU compat functions now, frame pools for preallocation, a private copy of engine helpers in the compat library).
• AGENTS.md invariant note — new core/src/AGENTS.d/vmafx-compat.md, updated core/src/AGENTS.d/vmafx-api.md, index entry in docs/development/rebase-sensitive-invariants.md; indexes regenerated.
• Reproducer / smoke-test command — below.
• CHANGELOG fragment — changelog.d/added/api-libvmaf-compat-library.md.
• Rebase note — docs/rebase-notes.d/vmafx-libvmaf-compat-split.md (fragment, ADR-2197), "libvmaf compat library on libvmafx: engine names, split library targets".

User documentation: docs/api/vmafx/index.md "Migrating from libvmaf.h" (layout, bridge, deprecation warnings, the deliberate differences, backend exceptions); generated docs/api/vmafx/compat.md (all 107 functions, the VMAFx calls each is written on, kind, library); docs/api/index.md; docs/development/upstream-consumers.md; docs/development/api-generation.md (the [[compat]] table and outputs); generated reference pages for the new headers.

Reproducer

python3 scripts/codegen/vmafx-api.py --check
python3 scripts/codegen/vmafx-api.py --abi-check --against-ref origin/master
meson setup build core -Denable_dnn=disabled -Db_lto=false
nice -n 10 ninja -C build -j4
python3 scripts/ci/run_meson_test.py -- -C build test_compat_conformance \
    test_compat_conformance_planted_score test_compat_conformance_planted_uncovered \
    check_exported_symbols check_exported_symbols_libvmaf test_compat_library_gates \
    test_libvmaf_deprecation test_vmafx_api_generator
nm -D --defined-only build/src/libvmafx.so.1 | grep -c ' vmaf_'      # 0
nm -D --defined-only build/src/libvmaf.so.3 | grep -c ' T vmaf_'     # 74 (82 with -Denable_mcp=true)
meson setup _inst core --prefix "$PWD/_prefix" -Denable_dnn=disabled -Db_lto=false
ninja -C _inst -j4 install
scripts/ci/upstream-ffmpeg-compat.sh --prefix "$PWD/_prefix" --against-cli
scripts/ci/upstream-gstreamer-compat.sh --prefix "$PWD/_prefix" --against-cli

Tests

Test	Covers
test_compat_conformance (fast)	12 scenarios (lifecycle, dictionary, models, collections, scoring over 3 frames with every score and pooled score as %a and every report format, preallocated pictures, pictures, conversion, perceptual weighting, tiny-AI with a real session on model/tiny/smoke_v0.onnx in DNN builds, HIP / Metal absent, MCP) run through the old libvmaf bodies and through libvmaf.so.3; traces must be equal and every compat function of the build must have been called (74 in a CPU build, 82 with MCP); the 25 engine exceptions are listed as such
test_compat_conformance_planted_score, _planted_uncovered (fast, should_fail)	one score of the compat side one ulp off; one scenario left out
check_exported_symbols, check_exported_symbols_libvmaf (fast)	libvmafx.so.1 exports exactly the vmafx_* list in its version nodes plus the built backends' legacy nodes, no vmaf_ otherwise; libvmaf.so.3 exports exactly the libvmaf functions the generated list gives this build, unversioned
test_compat_library_gates (fast)	a library calling an exported vmafx_* function links against libvmafx.so.1 under --no-undefined; one reaching vmaf_engine_init or vmaf_init does not; each export check fails on its list with a row removed
test_libvmaf_deprecation (fast)	every exported declaration names the successor the definition gives it; opt-in warning names vmafx_version_string; a default build and VMAF_BUILDING_LIBVMAF compile clean under -Werror; planted missing / wrong / unlisted markers found
test_vmafx_api_compat.py (15)	[[compat]] rules (unknown kind, engine without backend, exception without end, unknown backend, manual source outside the compat directory, undeclared call), engine-name header, legacy maps, symbol list rows, conformance tables, manual sources call exactly their declared vmafx_* functions (planted mismatch found), checker on planted missing, extra and versioned exports
test_upstream_consumer_scores.py (10)	the score comparison: identical files pass; one ulp, a trailing zero, a missing frame or metric fail; setup errors
test_gpu_public_header_docs (fast)	the Doxygen attachment check accepts the deprecation marker between block and declaration and still refuses other text

Gates shown failing on planted defects (measured on this branch)

Planted defect	Gate	Result
compat vmaf_score_at_index one ulp off	test_compat_conformance (VMAF_COMPAT_PLANT=score)	scenario scoring: traces differ at line 38, exit 1
scenario pictures skipped	test_compat_conformance (VMAF_COMPAT_PLANT=uncovered)	no conformance case calls vmaf_picture2_alloc (and 4 more), exit 1
vmaf_init row removed from the compat list	check_exported_symbols --library vmaf	exit 1
vmafx_submit row removed from the VMAFx list	check_exported_symbols --library vmafx	exit 1
a compat source calls an engine symbol	link under --no-undefined	undefined reference to vmaf_engine_init
marker missing / naming another target / unlisted function	test_libvmaf_deprecation	the three findings named
manual source calls an undeclared vmafx_* function	test_vmafx_api_compat.py	a.c: calls vmafx_y, no entry declares it
definition rules (kind, backend, end, source path, unknown call)	test_vmafx_api_compat.py	each refused with its message
a compat return value differed (missing model path -1 vs -22, overriding an extractor the model lacks)	test_compat_conformance, golden gate	both found during development and fixed in core/src/compat/libvmaf/model.c
vmaf_set_sample_range_check_enabled and vmaf_set_input_colorimetry declared in the headers without entries	test_libvmaf_deprecation	exported but not in core/api/vmafx.toml [[compat]] for both
the three conformance tests sharing one report file (found by a parallel run: one run read another's XML as its JSON)	test_compat_conformance	traces differ at line 162; fixed in f324808b4 (one file per run)

Upstream consumers (this host, CPU)

• FFmpeg n9.0.2, unpatched, libvmaf filter, 3 frames: built against the compat prefix; IDENTICAL to the same binary on the base library and to the CLI (45 values, PASS).
• GStreamer gst-plugins-bad 1.28.7 vmaf element, unpatched: IDENTICAL to the base library and the CLI (45 values, PASS).
• FFmpeg libvmaf_cuda: not run (no CUDA device on this host; the leg prints SKIP and never passes).

Builds measured

CPU (-Denable_dnn=disabled -Db_lto=false); DNN (-Denable_dnn=enabled, ONNX Runtime): conformance equal including real session runs; MCP (-Denable_mcp=true, every transport): 82 exports, conformance equal. GPU builds not measured here (device lanes own them); Windows .def linking and the darwin export list are WP12.

Known follow-ups

• Each RC4 WP3 lane (CUDA feat(api): import CUDA device frames with event fences and GL textures (RC4 WP3, ADR-2023) #2277, SYCL, HIP, Metal) turns its backend's engine exceptions into compat functions on device frames, deletes its legacy map and adds device conformance.
• WP12: release artifacts (scripts/release/build-native-release-artifacts.sh, supply-chain.yml) must stage libvmafx.so*; install tests; Windows .def; macOS export list.
• Two libvmaf functions are newer than this branch's base: vmaf_set_sample_range_check_enabled() (feat(api): add an opt-in check that refuses samples above 2^bpc - 1 #2221, already on master) and vmaf_set_input_colorimetry() (port(upstream): HDR-VMAF groundwork: input colorimetry, model conversion_target, conversion in read_pictures (#1671-#1675, #1677, #1678) #2300, ADR-2093, in the train). With them the libvmaf surface is 109 functions. Both are compat functions on this branch (done on the restack onto master); the colorimetry maps onto colour carried by VMAFx frames (VmafxFrameDesc grows a VmafxColor at its end, a context function sets the colour of frames that carry none, the submit hands each pair's colour to the engine's conversion state, VMAFX_E_BUSY as libvmaf's -EBUSY), with a conformance scenario on a conversion_target model (ADR-2094 follow-ups). test_libvmaf_deprecation refuses a header function without an entry: with both declarations planted into this branch's headers it fails naming both.