Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
82 changes: 82 additions & 0 deletions docs/adr/1122-vmaf-v1-model-port.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
<!-- markdownlint-disable MD013 MD041 MD060 -->

# ADR-1122: Adopt and port VMAF v1 models (opt-in, v0.6.1 stays default)

- **Status**: Proposed
- **Date**: 2026-07-01
- **Deciders**: Lusoris Dev (<dsp@mvdnet.org>)
- **Tags**: `vmaf`, `models`, `upstream-port`, `cambi`, `chroma`, `sycl`, `golden-gate`

## Context

Netflix shipped **VMAF v1** in libvmaf `v3.2.0` (2026-06-20) — a new generation
of models (`model/vmaf_v1.0.16/`). It is still a libsvm ν-SVR fusion, but the
**feature set changed**: VIF is removed, and CAMBI (banding) plus a chroma
feature (`speed_chroma_uv`) become core features; ADM/motion move to the
`adm3`/`motion3` variants. v1 adds viewing-condition-calibrated models (`1080p@3H`,
`phone@5H`, `4K@1.5H`, `4K@3H`), an `_hfr` high-frame-rate variant, a **[0, 110]**
score range for 4K@3H, CAMBI encode-side parameters, and a recommendation to
measure at **10-bit** even for 8-bit SDR. Details and citations are in the
[research digest](../research/2026-07-01-vmaf-v1-models.md).

The fork tracks Netflix/vmaf and already implements almost all v1 features on
GPU/SIMD (`adm3`, `motion3`, CAMBI, `speed_chroma`). Two of them — `cambi_sycl`
and `speed_chroma_sycl` — currently **fail on SYCL** (CAMBI parity failure;
speed_chroma needs fp64 and SIGABRTs on Arc A380). Under v1 these are no longer
optional: they are evaluated for **every** score. The fork must decide how far
to go, and whether v1 replaces or coexists with v0.

## Decision

We will **adopt VMAF v1 as a supported, opt-in model generation, keeping
`vmaf_v0.6.1` as the default.** Concretely:

1. **Port and build in the six v1 JSONs** (`vmaf_v1.0.16_{3d0h,5d0h,1d5h_2160,
3d0h_2160}` + the two `_hfr` variants) via `/add-model`, matching upstream's
built-in table. Selectable via `--model version=vmaf_v1.0.16_3d0h` etc.
2. **Fix the SYCL core features v1 depends on** — `cambi_sycl` parity and
`speed_chroma_sycl` (fp64-free per ADR-0220) — as a hard prerequisite, since
both are core v1 features on every backend.
3. **Support the [0, 110] range** end-to-end (CLI, `--precision`, the FFmpeg
`libvmaf*` filters and `vf_libvmaf` patches) — no clamp-to-100.
4. **Plumb the CAMBI encode-side params** (`cambi.enc_width/enc_height/
enc_bitdepth`) through the model-option parser and the FFmpeg filter AVOptions.
5. **Add v1 correctness tests against new v1 reference values** in *separate*
test files. The three Netflix v0.6.1 CPU golden pairs and their assertions
remain **untouched** (CLAUDE.md rule 1).
6. **Keep VIF** for v0 compatibility; the v1 dispatch simply does not register
it (off the v1 critical path).

The default model stays `vmaf_v0.6.1` for one release cycle to preserve the
golden gate and score-scale stability; v1 is opt-in until its GPU features and
tests are green across CPU/CUDA/SYCL/HIP.

## Alternatives considered

| Option | Pros | Cons | Why not chosen |
|---|---|---|---|
| **Port v1 opt-in, v0 stays default (chosen)** | Tracks upstream; no golden-gate/score-scale disruption; ships when GPU features are green | Two model generations to maintain for a while | Selected — correctness-preserving, incremental |
| Adopt v1 as the new default (replace v0.6.1) | Single generation; latest accuracy by default | Breaks the v0 golden gate; changes the score scale ([0,110]); silently shifts every downstream number | Rejected — violates the golden-data contract and surprises users |
| Don't port v1 (stay on v0) | No work | Falls behind Netflix; misses banding/chroma/phone accuracy; the fork's raison d'être is tracking upstream | Rejected — regressive |
| Port the JSONs only, defer SYCL CAMBI/chroma fixes | Fast to land models | v1 on SYCL would be broken (CAMBI/chroma are core, currently failing) — a shipped-but-broken backend | Rejected — v1 must work on the fork's flagship SYCL backend |

## Consequences

- **Positive**: the fork gains Netflix's latest, more-accurate, faster metric
(VIF removed), with banding/chroma awareness and viewing-condition models; the
10-bit recommendation aligns with the zero-copy P010 work (ADR-1121, PR #1081).
- **Negative**: two model generations coexist; the SYCL CAMBI/`speed_chroma`
fixes are now on the critical path (they were tolerated as failing because v0
doesn't fuse them). The `[0,110]` range touches user-visible output surfaces.
- **Neutral / follow-ups**: a `/sync-upstream` (or targeted `/port-upstream-commit`)
against `Netflix/vmaf v3.2.0` for the fusion/VIF-removal changes; new v1 golden
references; CAMBI enc-param AVOptions imply an ffmpeg-patch update (CLAUDE.md
rule 14); HDR v1 (Netflix "planned") is deferred/scaffolded.

## References

- req: user request (2026-07-01) — "исследуй новые модели v1 … заведи ADR на порт v1".
- Research digest: [docs/research/2026-07-01-vmaf-v1-models.md](../research/2026-07-01-vmaf-v1-models.md).
- Upstream: Netflix/vmaf `v3.2.0`; `resource/doc/models_v1.md`; `model/vmaf_v1.0.16/*.json`.
- Related: [ADR-1121](1121-sycl-qsv-zerocopy-p010-normalization.md) (10-bit zero-copy),
[ADR-0220](0220-sycl-fp64-fallback.md) (fp64-free SYCL kernels — blocks `speed_chroma_sycl`).
1 change: 1 addition & 0 deletions docs/adr/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -969,3 +969,4 @@ ADRs may exist there for local session continuity, but the tracked
| [ADR-1118](1118-perceptual-sidedata-weighting.md) | Pelorus side-data perceptually re-weights VMAF **spatial pooling** (opt-in, golden-isolated). The vendored interop parser (ADR-1113) reads each frame's per-cell banding-risk / variance maps; a normalized `[0,1]` salience becomes a per-frame pooling weight `w = 1 + strength·salience`, turning pooled MEAN/HARMONIC_MEAN into their weighted forms (MIN/MAX unaffected). **Golden-gate isolation (load-bearing):** inert unless BOTH the `perceptual_weight` opt-in is set AND a valid Pelorus blob is present for the frame; otherwise `w ≡ 1.0` and the pooling runs the *literal* upstream expression, so the no-side-data path (and the Netflix golden pairs, which carry no side-data) is byte-identical — proven by `test_perceptual_weight.c`. New C-API `vmaf_set_perceptual_{weight_enabled,weight_strength,sidedata}` (`core/include/libvmaf/perceptual_weight.h`); weight module `core/src/feature/perceptual_weight.c`; `vf_libvmaf` reader + `perceptual_weight` AVOption (ffmpeg-patch 0017). R1–R6 compat: `min(known_size,dir.size)` reads, unknown bits ignored, `grid==0` degrades to frame-level scalar, ABI-major mismatch rejected (unweighted + log). CPU-only, no GPU. Alternatives (auxiliary-feature / both) rejected — maintainer chose spatial-pooling weighting. | Accepted | 2026-06-14 | scoring, pooling, pelorus, interop, ffmpeg, golden-gate, fork-local |
| [ADR-1119](1119-golusoris-go-framework-adoption.md) | Adopt the `github.com/golusoris/golusoris` fx framework (pinned `v0.4.0`) across all six vmafx Go binaries + `pkg/`. Every binary becomes an `fx.New(...).Run()` over golusoris modules (Core/otel/HTTP/grpc/k8s-operator/clikit), sharing a vmafx-local `internal/app/bootstrap` stanza (`Base` + `FxLogger`). RC-blocking, phased services-first (`vmafx-server` → `controller` → `node` → `operator`, then `mcp`/`tune` + `pkg` sweep). Keeps the `VMAFX_` config env-prefix via `fx.Replace(config.Options{...})` (not golusoris's default `APP_`) and keeps the controller's embedded SQLite queue (not `golusoris.Jobs`/Postgres). Dependency closures already align byte-for-byte, so the `go.mod` merge is low-risk (`go build ./...` clean post-`go get`). Four framework gaps filed upstream were all integrated by the maintainer (#225 gRPC interceptor injection, #226 version module, #227 operator SetLogger + webhook, #234 log reads `log.level` from config); the pin was bumped v0.3.1→v0.4.0. Only #226 + the `k8s/operator` module are in the v0.4.0 tag; #225/#227/#234 are merged to golusoris main but untagged, so the service binaries carry small interim shims (the #234 log-env bridge; the #227 operator SetLogger + webhook gate) and the controller (needs #225) follows the next golusoris tag. | Accepted | 2026-06-14 | go, framework, fx, golusoris, server, controller, node, operator, mcp, vmaf-tune, rc-blocking, fork-local |
| [ADR-1120](1120-pelorus-abi-minor3-resync-complexity.md) | Re-pin the vendored Pelorus interop ABI (ADR-1113) from `pelorus@835e097` (ABI 1.0) to `pelorus@818d844` (ABI 1.3) and consume the new `PEL_SEC_COMPLEXITY` per-frame section in perceptual weighting (ADR-1118). Minor-3 appends three sections (`QPREPORT` 1.1, `MOTION_CONF` 1.2, `COMPLEXITY` 1.3) and three files; the mirror gains `denoise.h`, `denoise_params.c`, `qp_report_csv.c` (the last is required for the minor-3 conformance fixture to link `pel_x265_csv_parse`). The `PelorusSideData` header layout is unchanged (append-only R1) and the parser keys on `abi_major`, so it is a deliberate re-pin, not a break. Fixes the sync-script `--update` bug that re-vendored the six manifest files but not the conformance-fixture body (the immediately-following drift check then failed). **Complexity modulation:** `perceptual_weight.c` attenuates the banding salience by `(1 − 0.5·complexity)` floored at `0.25` — banding is more visible on flat/simple frames and masked on busy/textured frames. **Golden-isolated (load-bearing):** engages only when the opt-in is set AND a complexity section is present; absent section → factor `1.0` → byte-identical pooling, so the Netflix golden 576×324 pair scores `76.667831` unchanged (verified). Proven by `test_complexity_modulates_weight` / `_grid_zero` (toggle-proven). New vendored files inherit the existing prefix-glob lint/format exclusions. CPU-only, no GPU. Alternatives (trim-fixture, threshold/step modulation, replace-banding) rejected. | Accepted | 2026-06-27 | interop, abi, vendoring, pelorus, scoring, pooling, golden-gate, build, fork-local |
| [ADR-1122](1122-vmaf-v1-model-port.md) | Adopt & port **VMAF v1** models (Netflix, libvmaf v3.2.0, June 2026) as an **opt-in** generation while **`vmaf_v0.6.1` stays default**. v1 is still a libsvm ν-SVR fusion but the feature set changed: **VIF removed**, **CAMBI (banding) + `speed_chroma` (chroma) promoted to core**, ADM/motion → `adm3`/`motion3`. Six built-in JSONs (`vmaf_v1.0.16_{3d0h,5d0h,1d5h_2160,3d0h_2160}` + two `_hfr`) for 1080p@3H / phone@5H / 4K@1.5H / 4K@3H (the last on a **[0, 110]** range). Netflix recommends **10-bit** measurement even for 8-bit SDR (banding) — aligns with the zero-copy P010 path (ADR-1121). Port scope: build in the JSONs (`/add-model`); **fix SYCL `cambi`/`speed_chroma`** (now core, currently failing on Arc A380 — fp64 SIGABRT / parity fail); handle `[0,110]` end-to-end (CLI/`--precision`/ffmpeg filters, no clamp-to-100); plumb CAMBI `enc_width/enc_height/enc_bitdepth` (ffmpeg-patch impact); add **new** v1 golden refs in separate tests (the 3 Netflix v0.6.1 golden pairs stay untouched). Alternatives (v1-as-default / don't-port / models-only) rejected — golden-gate + score-scale + broken-SYCL respectively. | Proposed | 2026-07-01 | vmaf, models, upstream-port, cambi, chroma, sycl, golden-gate, fork-local |
1 change: 1 addition & 0 deletions docs/adr/_index_fragments/1122-vmaf-v1-model-port.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
| [ADR-1122](1122-vmaf-v1-model-port.md) | Adopt & port **VMAF v1** models (Netflix, libvmaf v3.2.0, June 2026) as an **opt-in** generation while **`vmaf_v0.6.1` stays default**. v1 is still a libsvm ν-SVR fusion but the feature set changed: **VIF removed**, **CAMBI (banding) + `speed_chroma` (chroma) promoted to core**, ADM/motion → `adm3`/`motion3`. Six built-in JSONs (`vmaf_v1.0.16_{3d0h,5d0h,1d5h_2160,3d0h_2160}` + two `_hfr`) for 1080p@3H / phone@5H / 4K@1.5H / 4K@3H (the last on a **[0, 110]** range). Netflix recommends **10-bit** measurement even for 8-bit SDR (banding) — aligns with the zero-copy P010 path (ADR-1121). Port scope: build in the JSONs (`/add-model`); **fix SYCL `cambi`/`speed_chroma`** (now core, currently failing on Arc A380 — fp64 SIGABRT / parity fail); handle `[0,110]` end-to-end (CLI/`--precision`/ffmpeg filters, no clamp-to-100); plumb CAMBI `enc_width/enc_height/enc_bitdepth` (ffmpeg-patch impact); add **new** v1 golden refs in separate tests (the 3 Netflix v0.6.1 golden pairs stay untouched). Alternatives (v1-as-default / don't-port / models-only) rejected — golden-gate + score-scale + broken-SYCL respectively. | Proposed | 2026-07-01 | vmaf, models, upstream-port, cambi, chroma, sycl, golden-gate, fork-local |
1 change: 1 addition & 0 deletions docs/adr/_index_fragments/_order.txt
Original file line number Diff line number Diff line change
Expand Up @@ -878,3 +878,4 @@
1118-perceptual-sidedata-weighting
1119-golusoris-go-framework-adoption
1120-pelorus-abi-minor3-resync-complexity
1122-vmaf-v1-model-port
141 changes: 141 additions & 0 deletions docs/research/2026-07-01-vmaf-v1-models.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,141 @@
# VMAF v1 Models — Research Digest

**Date:** 2026-07-01
**Author:** VMAFx maintainers
**Upstream:** Netflix/vmaf `v3.2.0` (2026-06-20)
**ADR:** [ADR-1122](../adr/1122-vmaf-v1-model-port.md)
**Sources:** Netflix TechBlog "VMAF v1: Good Is Not Good Enough" (2026-06);
`Netflix/vmaf` `resource/doc/models_v1.md`; `model/vmaf_v1.0.16/*.json`;
`libvmaf/src/model.c` (built-in model table).

---

## Summary

Netflix shipped a new generation of VMAF models — **VMAF v1** — in libvmaf
`v3.2.0` (June 2026). v1 is **not** a new fusion architecture: it is still a
libsvm ν-SVR (`model_type: LIBSVMNUSVR`) over quality-aware features. What
changed is the **feature set** and the **model calibration** for viewing
conditions. The headline change is that **VIF is removed** and **CAMBI
(banding) + a chroma feature are promoted to core features**. Netflix also
recommends measuring at **10-bit precision** even for 8-bit SDR content, adds a
4K model with a **[0, 110]** score range, and provides high-frame-rate (`_hfr`)
variants.

This digest records the exact model set, the feature delta, and the concrete
implications for the fork (which already implements most of the v1 features on
GPU). It backs [ADR-1122](../adr/1122-vmaf-v1-model-port.md).

---

## The v1 model set (`model/vmaf_v1.0.16/`)

All models are libsvm ν-SVR JSONs, built into libvmaf (`model.c` exposes
`src_vmaf_v1_0_16_*_json`), selectable via `--model version=…` or
`--model path=…json`.

| Scenario | Display | Viewing distance | Model file | Score range |
| --- | --- | --- | --- | --- |
| Standard 1080p | 1080p | 3H | `vmaf_v1.0.16_3d0h.json` | [0, 100] |
| Phone | 1080p | 5H | `vmaf_v1.0.16_5d0h.json` | [0, 100] |
| **4K (default)** | 2160p | 1.5H | `vmaf_v1.0.16_1d5h_2160.json` | [0, 100] |
| 4K consumer TV | 2160p | 3H | `vmaf_v1.0.16_3d0h_2160.json` | **[0, 110]** |

Plus `_hfr` variants under `model/vmaf_v1.0.16_hfr/` (built-in as
`src_vmaf_v1_0_16_hfr_3d0h_json`, `…_hfr_3d0h_2160_json`) calibrated for
~50/60 fps: a **five-frame temporal motion window** (differencing over frames
i−2, i, i+2) with moving-average smoothing, correcting v0's under-prediction at
high frame rates.

## Feature delta (the core change)

The v1 standard model's `feature_names` (from `vmaf_v1.0.16_3d0h.json`):

```text
Cambi_feature_cambi_score
Speed_chroma_feature_speed_chroma_uv_score
VMAF_integer_feature_adm3_score
VMAF_integer_feature_motion3_score
```

| | v0 (`vmaf_v0.6.1`) | **v1 (`vmaf_v1.0.16`)** |
| --- | --- | --- |
| features | adm2, **vif_scale0..3**, motion2 | **cambi**, **speed_chroma_uv**, **adm3**, **motion3** |
| count | 6 | **4** |
| VIF | 4 scales (dominant cost) | **removed** |
| banding | separate CAMBI metric | **core feature** |
| chroma | — | **core feature** (`speed_chroma`) |

- **VIF removed.** Netflix found VIF is the most expensive feature and, after
refining the others, dropped it without accuracy loss — so v1 is **faster**.
- **CAMBI in the fusion.** Banding (previously a standalone metric) is now a
VMAF feature, addressing v0's blindness to banding.
- **Chroma feature added** (`speed_chroma_uv`) — v1 accounts for chroma
distortion, which v0 ignored (luma-only).
- **ADM → `adm3`, Motion → `motion3`** (specific variants; motion3 is the
wider temporal window used more aggressively by the `_hfr` variants).

## Key behavioral changes

1. **10-bit is the recommended precision.** *"VMAF v1 should ideally be applied
at 10-bit precision for SDR"* (models_v1.md) — even 8-bit encodes should be
measured at 10-bit (preprocess both inputs) so CAMBI can resolve banding.
2. **Scores can exceed 100.** The 4K@3H model uses **[0, 110]** to quantify the
4K-over-1080p benefit at 3H. Any `min(score, 100)` assumption is now wrong.
3. **CAMBI encode-side params.** `--model …:cambi.enc_width=W:cambi.enc_height=H:
cambi.enc_bitdepth=B` passes the *pre-upscale* encode geometry to the CAMBI
feature; merged into the model's CAMBI options (no separate CAMBI instance).
Falls back to input W/H/bitdepth when unset.
4. **Selection unchanged.** Same `--model version=…` / `path=…` mechanism as v0;
v1 and v0 coexist (v0 preserved under `models_v0.md`).

## Performance & accuracy (Netflix's claims)

- v1 **matches or outperforms** v0 on most datasets; notable gains on WATERLOO
IVC 4K, Netflix Screen-Size Crowdsourcing, and datasets with chroma/banding
artifacts and phone viewing.
- **Faster** than v0 (VIF — the compute bottleneck — is gone).
- Future: a detailed technical paper and an **HDR v1** model are planned.

## Implications for the fork (VMAFx)

The fork already implements **almost every v1 feature on GPU/SIMD**:

| v1 feature | Fork status |
| --- | --- |
| `adm3` | ADM extractor present (CUDA/SYCL/HIP); adm scales incl. scale 3 |
| `motion3` | present (`integer_motion3` column already emitted) |
| `cambi` | present (`cambi_sycl`, `integer_cambi_*`); **SYCL parity currently FAILING** (`test_sycl_cambi_parity` exit 1 — see Phase-3 verify) |
| `speed_chroma` | present (`speed_chroma_*`); **SYCL FAILS on Arc A380** (`test_sycl_speed_chroma_parity` SIGABRT: fp64-not-supported) |
| VIF | present but **no longer on the v1 critical path** |

Net: v1 is a **medium port**, not a rewrite. The load-bearing work:

1. **Port + build-in the 6 v1 JSONs** (`/add-model`), matching Netflix's
built-in table.
2. **Fix SYCL CAMBI + speed_chroma** — these become *core* features for every
v1 score. `speed_chroma_sycl` currently needs fp64 (SIGABRT on Arc A380,
ADR-0220 territory); `cambi_sycl` parity fails. These are **v1 blockers on
SYCL**.
3. **Handle the [0, 110] range** — audit CLI/filters/`--precision` and any
clamp-to-100. The FFmpeg `libvmaf*` filters and `vf_libvmaf` patches must not
truncate.
4. **VIF is now optional for v1** — keep for v0 compatibility, but the v1 path
skips it (potential GPU speedup: the fork's whole VIF kernel stack is off the
v1 critical path).
5. **10-bit alignment.** v1's 10-bit recommendation directly validates the
zero-copy P010 path (ADR-1121, PR #1081) — the fork is already positioned to
measure v1 at the recommended precision, unlike 8-bit tooling (e.g. FFMetrics).
6. **Golden gate.** The 3 Netflix CPU golden pairs are v0.6.1 — **do not touch**
(CLAUDE.md rule 1). v1 needs **new** golden references in *separate* tests.
7. **Upstream sync.** `Netflix/vmaf v3.2.0` is the port target
(`/sync-upstream` / `/port-upstream-commit`) — large: new models + VIF
removal from fusion + CAMBI/chroma integration + CAMBI enc-params.

## Open questions for the port

- Does `adm3`/`motion3` in the fork bit-match upstream v1's feature definitions,
or did upstream change the integer kernels for v1? (Verify against v3.2.0.)
- CAMBI enc-param plumbing through the FFmpeg filter AVOptions (ffmpeg-patch
impact under CLAUDE.md rule 14).
- HDR v1 timing (Netflix "planned") — defer or scaffold.