Repository navigation
refactor(interop): re-vendor Pelorus at the commit that opens the qp-report CSV with _wfsopen (ADR-1113) - #2424
Merged
Merged
Conversation
…y tester leg before a cut (ADR-2198) (#2409) * ci: build the Windows SYCL zip where its inputs change and check every tester leg before a cut (ADR-2198) The x64 SYCL tester zip was red on master for two days with nothing required noticing. A pull request now builds it when an input it reads changes, in the light tier through own_input_lanes, and the cut pull request runs scripts/release/check-candidate-legs.py. * docs: regenerate the indexes and the citation map after rebasing
* feat(rust): add the RC4 Rust extractor framework crates and the psnr reference twin
The workspace gains vmafx-fex (the ABI of the Rust twins, the Extractor
trait, option, picture and host views, libm wrappers), the reference twin
vmafx-fex-psnr, empty crates for the speed_chroma, motion, adm, cambi and
prediction lanes, and vmafx-core-rs, the one Rust archive libvmaf links.
The TAD pilot becomes an rlib of that archive and drops its unused
build-time cbindgen dependency, so the Rust build needs no network. The C
header of the ABI is generated by scripts/dev/rust-abi-header.sh and
committed. ADR-1713.
* feat(rust): register the Rust twins through a C shim and select them with VMAF_FEATURE_IMPL
core/src/rust/shim/rust_twins.cpp turns each Rust twin into a
VmafFeatureExtractor named <c name>_rust that copies the C extractor's
descriptor (options, priv size, provided features, flags) and drives the
Rust entry points with the parsed options, plane views, the previous
references and the C feature collector. feature_extractor.cpp walks the
static list and then the Rust extractors the shim installs at vmaf_init(),
skips Rust twins in plain feature-name lookups, audits them like device
twins and offers vmaf_feature_extractor_impl_select(), which libvmaf
applies on every registration path when VMAF_FEATURE_IMPL=rust.
Meson builds one archive (vmafx-core-rs) with an offline cargo build and
a depfile, links it into libvmaf only, and keeps its symbols out of the
dynamic symbol table with --exclude-libs. The TAD pilot is now registered
in Rust builds; before, feature_extractor.cpp never saw HAVE_RUST_TAD.
scripts/ci/rust_twin_diff.py runs both implementations of one binary and
requires equal doubles on every metric and a receipt naming the twin; the
psnr reference twin is equal on the Netflix pair, both checkerboard pairs,
the 10-bit sparks pair and 200 frames of 4K. ADR-1713.
* feat(rust): give the Rust twins a host log and a log-only close
Lane request S-1: VmafxRsHost gains a `log` callback (appended, so no
offset moves) and the twin's close entry point receives a host whose
collector callbacks fail but whose log works, so a twin can report what
its C extractor logs at close. Extractor::close() runs before the state
drops; Host::log_fmt formats into a stack buffer without allocating.
Lane request P-1: the header script also generates
core/src/rust/include/vmafx_rs_predict.h once the predictor's cbindgen
configuration exists, and vmaf_feature_impl_rust_requested() is the one
reader of VMAF_FEATURE_IMPL for every Rust path.
The build wrapper keeps a plain subprocess call: the safe_subprocess
helper resolves symlinks and a rustup cargo is a symlink to rustup. The
shim asserts its preconditions (Power-of-10 rule 5) and propagates the
dictionary free result instead of discarding it.
* docs(rust): document the Rust extractor framework and gate it in the Rust workflow
docs/development/rust-extractor-framework.md explains how to build and
select the Rust path, how a twin is registered and written, the arithmetic
rules for bit identity, the differential harness and its fixtures, and the
tests. VMAF_FEATURE_IMPL joins the environment variable reference; the
build-flag, Rust guide and TAD pages follow the new build. A core/src
AGENTS.d page records the invariants of the framework, the TAD crate's
AGENTS.md its new shape.
The Rust workflow runs fmt and clippy on every workspace crate, checks the
cbindgen header, builds libvmaf with the Rust extractors, runs the rust
Meson suite and the harness on the Netflix and checkerboard pairs; the
impact selector covers the new paths and the Meson runner inventory lists
the workflow. docs/state.md records the two defects fixed on the way and
the missing container toolchain.
* fix(rust): format the host text buffer and record the psnr twin's provenance
Lane request C-1: cargo fmt --all --check failed on the StackText literal
in vmafx-fex's host.rs. Lane request A-1: the reference twin ports
integer_psnr.c statement by statement, so relicense_provenance.toml gives
it the netflix-integer-psnr family (documented-port instead of
provenance-unreviewed); each RC4 lane adds the block of its own crate.
* build(rust): give the libvmaf-linked crates a workspace without external crates
An offline cargo build from an empty CARGO_HOME failed in the root
workspace ("failed to select a version" for bindgen, required by
vmafx-sys): cargo resolves the whole workspace even for one package. The
framework crates, the twins, the predictor, vmafx-core-rs and the TAD
crate now form core/src/rust/Cargo.toml, whose lockfile holds only those
nine crates; the root workspace keeps the bindings and excludes both
directories. Meson builds against the new manifest, and the Rust workflow
runs fmt, clippy and tests on both workspaces and builds vmafx-core-rs
offline from an empty CARGO_HOME, which fails as soon as a crate there
gains an external dependency. The TAD rebuild step went with TAD's
build.rs.
* fix(rust): initialise a Rust twin's shared context before a threaded flush
Lane request M-1: with --threads N, flush_non_temporal_cpu_extractors()
calls flush() on the shared context of a non-temporal extractor, which is
never initialised (only the per-thread copies are). The C extractors'
flush needs no init state; a Rust twin's flush found no instance and the
run failed with "problem flushing context". init_shared_rust_twin()
initialises that context with the run's picture parameters first. Lane M
measured motion_rust with the five-frame window equal to C at --threads 1
and 4 with the change, failing without it.
Lane request A-2: the harness runs every cell once per --threads value
(default 0,1), and a cell where both sides refuse the input with the same
exit status is REFUSED and passes (speed_chroma at prescale 0.5 refuses
480x270 in C), while one side refusing alone stays an error.
* test(rust): run every twin cell serially and with thread pools of 1 and 4
The flush of a non-temporal extractor runs on the shared context only with
a thread pool, so the harness default becomes --threads 0,1,4 and the rust
Meson suite's harness test runs all three. motion and motion_v2 are the only
non-temporal C extractors with a flush; motion_rust (lane M) is the twin
that exercises init_shared_rust_twin() and failed at --threads 1 and 4
without it.
* docs(rust): give each RC4 twin its own row in the framework guide
Lane request A-3: one row per twin, so the PR that lands a twin edits only
its own row.
* fix(rust): hold the framework to the licence and tidy-coverage gates master gained
Two gates landed on master after the framework branched:
- The package licence check (ADR-1699) wants each crate's `license` to be
the AND of the licences of the files it ships. Every crate of the
libvmaf-linked workspace ships the root README.md (BSD-2-Clause-Patent in
REUSE.toml) next to its EUPL-1.2 sources, and the psnr twin carries
statements ported from integer_psnr.c, so the workspace declares
"EUPL-1.2 AND BSD-2-Clause-Patent", as vmafx-tad already does.
- The clang-tidy coverage rule wants every tracked translation unit read by
a lane or named in an exception. The shim and its two tests build only
with -Denable_rust_features=true, and the dev image every lane runs in
has no Rust toolchain yet, so they get the entry tad_rust.c and
test_tad_rust.c already have, with the same expiry, until a pinned
toolchain is in dev/Containerfile and a lane reads them.
* docs(adr): accept ADR-1713 for the RC4 Rust extractor framework (Q-087)
The maintainer accepted the framework decision as written (ledger Q-087,
2026-10-07): Rust twins registered next to the C extractors as
`<name>_rust`, selected with VMAF_FEATURE_IMPL, one Rust archive, the C ABI
unchanged. The status line, the deciders and the index row say so, and the
References cite the ledger entry.
* fix(test): return the TAD test's failure message without discarding const
expect_score() took its message as `const char *` and returned it as the
`char *` the test runner expects, which GCC reports as
-Wdiscarded-qualifiers. The file builds only with
-Denable_rust_features=true, so no warning-checked leg compiled it before
the Rust framework gate. The message is a string literal like every other
message the runner returns; it is now passed as `char *`.
* test(build): follow the TAD pilot into the Rust framework in the stale-text contract
The contract that landed on master (#1976, defect 18) checks that a build
without enable_rust_features neither compiles tad_rust.c nor registers
TAD, and it spelled that through the pilot's own Meson list
(rust_tad_direct_sources) and its #if HAVE_RUST_TAD registry entry. The
framework (ADR-1713) replaced both: tad_rust.c is a direct libvmaf source
inside the Rust block next to the shim, and the shim registers the pilot.
The test now reads that shape: the source is listed once, inside the Rust
block, the shim assigns vmaf_fex_tad, and the static registry does not name
it. Moving the source out of the Rust block fails the test.
* fix(rust): give the VMAF_FEATURE_IMPL selector enum a one-byte underlying type
clang-tidy 22.1.8 (cpu lane, dev container) reported performance-enum-size on enum class FeatureImpl in core/src/feature/feature_extractor.cpp: three enumerators fit std::uint8_t. The lane now reads 0 findings in feature_extractor.cpp and libvmaf.c.
* build(docker): copy the Rust twin harness into the tester image's build stages
Master's tester image contract (tools/rc1-tester/tests/
test_dockerfile_script_imports.py) requires every Meson build stage to copy
each file outside core/ that the Meson tree names; the framework's rust
suite names scripts/ci/rust_twin_diff.py, so the vmaf, SYCL, CUDA and HIP
build stages copy it. The test failed without the four lines.
* docs(adr): regenerate the ADR index pages after the rebase onto master
* docs(adr): regenerate the generated files after the rebase onto master
* docs: regenerate the indexes and the citation map after rebasing
…report CSV with _wfsopen (ADR-1113) (#2424) * refactor(interop): re-vendor Pelorus at the commit that opens the qp-report CSV with _wfsopen (ADR-1113) VMAFx/pelorus #89 (fixing #88) moves open_utf8() from the deprecated _wfopen() to _wfsopen(..., _SH_DENYNO), the local edit the MSVC zero-warnings series carried in core/src/interop/pelorus_qp_report_csv.c. PELORUS_VENDOR_SHA moves to 4aae30711c65 and --update re-renders the ten vendored files; the mirror is byte-identical to pelorus again apart from the banner and the include rewrite, and the drift check passes.
8 of 11 tasks
lusoris
force-pushed
the
refactor/pelorus-revendor-wfsopen
branch
from
October 7, 2026 16:43
b588ab7 to
61c9016
Compare
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
BLUF: the vendored Pelorus mirror is byte-identical to pelorus again. The MSVC zero-warnings series (#2393) had to keep a local
_wfsopenedit incore/src/interop/pelorus_qp_report_csv.cto clear C4996; that fix is now pelorus's own code (VMAFx/pelorus #89, fixing VMAFx/pelorus#88), soPELORUS_VENDOR_SHAmoves to4aae30711c65andscripts/sync-pelorus-interop.sh --updatere-renders the ten vendored files._wfsopen(..., _SH_DENYNO)call; the code is unchanged against master.FAIL: vendored Pelorus interop ABI has drifted from pelorus@5f5614b0229d..., exit 1: the hostedLint / Pre-Commitfailure of run 37618697250) and passes on this head:OK: vendored Pelorus interop ABI matches pelorus@4aae30711c655510305e9c14b403f991d342f760 (ABI 1.3, minor=3).test_pelorus_interoppasses on a Linux CPU build.libpelorus (build · test · format · tidy)check passed).Type
refactor— no behaviour changeChecklist
Bug-status hygiene
Netflix golden-data gate
assertAlmostEqual(...)score in the Netflix golden Python tests.Deep-dive deliverables
AGENTS.mdinvariant note — no rebase-sensitive invariants beyond ADR-1113's existing mirror rule.changelog.d/changed/pelorus-revendor-wfsopen.md.docs/rebase-notes.md.Reproducer