Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
- **`fr_regressor_v2_ensemble_v1_seed{0..4}` registry rows: restored
dropped `license` / `license_url` / `sigstore_bundle` metadata and
made the smoke/production state honest.** PR #865 regenerated the five
ensemble ONNX files in smoke mode to fix a `codec_vocab` 14→6 input-dim
mismatch, but also dropped the per-row license metadata and left the
rows labelled neither production nor consistently smoke. The license
fields are restored, `smoke: true` now truthfully describes the shipped
weights, and the production flip is deferred to the one-shot post-RC
retrain (ADR-1105) with a strict-xfail tracking marker on
`test_fr_regressor_v2_ensemble_seed_rows_are_production`.
135 changes: 135 additions & 0 deletions docs/adr/1105-ensemble-v2-prod-flip-deferred-oneshot-retrain.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@
<!-- markdownlint-disable MD013 MD060 -->
# ADR-1105: `fr_regressor_v2_ensemble` production flip deferred to the one-shot post-RC retrain

- **Status**: Accepted
- **Date**: 2026-06-13
- **Deciders**: Lusoris
- **Tags**: `ai`, `models`, `rc`, `docs`

## Context

[ADR-0321](0321-fr-regressor-v2-ensemble-full-prod-flip.md) flipped the
five `fr_regressor_v2_ensemble_v1_seed{0..4}` rows in
`model/tiny/registry.json` from smoke to production: it shipped real
LOSO-validated ONNX weights (gate verdict `PROMOTE`, mean PLCC ≈ 0.997),
per-seed sidecars, and `smoke: false`. Those weights were trained with a
codec one-hot of width 14 (`codec_vocab` = 12 codec entries + 2 norm
dims).

`codec_vocab` was subsequently trimmed to 6 (`x264`, `x265`,
`libsvtav1`, `libvvenc`, `libvpx-vp9`, `unknown`; see
`model/tiny/fr_regressor_v2_ensemble_v1.json`). This made the
production ONNX input dimension stale: a model expecting `[batch, 14]`
can no longer be fed the current `[batch, 6]` codec one-hot, which
surfaced as an `eval_probabilistic_proxy.py --smoke` load failure.

PR #865 fixed the load path by regenerating the five ONNX files at the
correct `[batch, 6]` width, but it did so with the trainer's `--smoke`
mode — one epoch on a synthetic corpus, i.e. throwaway placeholder
weights, not a production fit. The registry was correspondingly set to
`smoke: true` with new sha256 values. Two side effects of that PR were
not intended:

1. The `license`, `license_url`, and `sigstore_bundle` fields were
dropped from the five rows. This is a pure regression — every
registry entry (smoke or not) must carry license metadata, and the
`test_every_entry_has_license_metadata` invariant enforces it. The
fields are restored unconditionally in this PR.
2. The five rows now claim neither production nor a consistent provenance
record: the on-disk ONNX are smoke (new sha), while the retained
per-seed sidecars still describe the older `[batch, 14]` production
weights (old sha, `PROMOTE`). The
`test_fr_regressor_v2_ensemble_seed_rows_are_production` invariant
(added by ADR-0321) consequently fails on `smoke is False`.

Producing real production weights at `codec_vocab = 6` requires
re-running `export_ensemble_v2_seeds.py` against the corpus — a full
retrain/re-export. The operator has locked all model retraining into a
single one-shot step to be run only after the toolchain reaches RC and
the feature numbers are frozen, explicitly to avoid retraining models
repeatedly. The ensemble is in scope for that one-shot retrain. Doing a
piecemeal ensemble-only retrain now would contradict that decision and
risk shifting numbers that the one-shot run is meant to freeze.

## Decision

For the release candidate, ship the ensemble seed rows honestly as smoke
placeholders and defer the production flip to the locked one-shot retrain:

1. Restore `license`, `license_url`, and `sigstore_bundle` on all five
rows (the #865 regression). Keep `smoke: true` and the regenerated
`[batch, 6]` sha256 values, which match the ONNX actually shipped.
Update each row's `notes` to state plainly that these are smoke
placeholders pending the one-shot production re-export.
2. Keep the `test_fr_regressor_v2_ensemble_seed_rows_are_production`
assertions verbatim (they remain the target production contract), but
mark the test `@pytest.mark.xfail(strict=True)` with a reason citing
this ADR. `strict=True` means the test fails the suite the moment the
one-shot retrain lands real weights (`smoke: false` + a sidecar whose
sha256 matches the shipped ONNX), forcing removal of the marker — so
the deferral cannot silently outlive its cause.
3. When the one-shot retrain runs, it must re-run
`export_ensemble_v2_seeds.py` so the ONNX bytes and sidecars
regenerate together at `codec_vocab = 6`, then flip `smoke: false` and
remove the xfail marker — exactly the workflow ADR-0321's follow-ups
already mandate (hand-flipping rows remains forbidden).

This does not modify any Netflix golden-data assertion (CLAUDE.md §8);
`model_registry_schema_test.py` is a fork-local file.

## Alternatives considered

- **Retrain the five ensemble seeds now (production flip immediately).**
This is the eventual correct end state but contradicts the locked
one-shot retrain decision (no piecemeal retraining before the
toolchain is RC-frozen) and would risk moving numbers the one-shot run
is meant to freeze. Rejected for RC; it is precisely what the one-shot
retrain will do.
- **Revert the ONNX to the old `[batch, 14]` production weights.**
Restores `smoke: false` consistency but re-breaks the load path under
the current `codec_vocab = 6`, reintroducing the
`eval_probabilistic_proxy` failure #865 fixed. Rejected.
- **Leave the test failing as a known local red.** The schema test is
non-gating in CI (it runs under the `|| true` block in
`tests-and-quality-gates.yml`), so this would not break master. But a
bare red is noise that can mask a future regression in the same test
and carries no self-healing signal. The strict-xfail marker is the
honest, self-documenting, auto-alerting representation. Rejected in
favour of strict xfail.
- **Delete the stale per-seed sidecars.** They describe the older
production weights, not the shipped smoke ONNX. Keeping them preserves
the genuine `PROMOTE` provenance and the sha the one-shot retrain will
supersede; the only consumer is the now-xfailed production test.
Rejected (kept) to retain provenance.

## Consequences

- **Positive**: The RC registry is internally consistent and honest —
every entry has license metadata, and `smoke: true` truthfully
describes the shipped weights. The deferral is tracked by a strict
marker that fails loudly when resolved.
- **Negative**: The probabilistic ensemble head ships at smoke quality in
the RC. Any consumer that loads it gets placeholder predictions until
the one-shot retrain. This is documented in the model card and state.md.
- **Neutral / follow-ups**: The one-shot retrain must (a) re-export the
five seeds via `export_ensemble_v2_seeds.py` at `codec_vocab = 6`,
(b) flip `smoke: false`, (c) remove the xfail marker in
`model_registry_schema_test.py`. Tracked in `docs/state.md` and the
retrain plan.

## Supply-chain impact

- **New dependencies**: none.
- **Removed dependencies**: none.
- **Build-time fetches**: none.

## References

- Parent / superseded-context: [ADR-0321](0321-fr-regressor-v2-ensemble-full-prod-flip.md),
[ADR-0303](0303-fr-regressor-v2-ensemble-prod-flip.md),
[ADR-0309](0309-fr-regressor-v2-ensemble-real-corpus-retrain.md).
- Regression source: PR #865 (`8b7ae731a`) — dropped license metadata and
regenerated ONNX in `--smoke` mode.
- `req` — operator direction: retrain all models exactly once, after the
toolchain reaches RC and feature numbers are frozen, to avoid repeating
the retrain; the ensemble is in scope for that one-shot run.
1 change: 1 addition & 0 deletions docs/adr/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -914,3 +914,4 @@ ADRs may exist there for local session continuity, but the tracked
| [ADR-1102](1102-phase4b9-container-only-publishing.md) | Phase 4b.9 container-only canonical artifact policy: the vmaf-dev-mcp container is the exclusive source for release binaries, published container images, and CI benchmark/snapshot artifacts. Host-side builds are diagnostic-only (IDE/clangd, debugger, sanitizer sweeps). Extends ADR-0496 to the publishing surface and adds docs/development/publishing.md. | Accepted | 2026-06-08 | container, build, release, publish, phase4b, docs-policy, fork-local |
| [ADR-1103](1103-hip-vif-mirror2-boundary.md) | Fix integer_vif_hip boundary condition: clamp_i → mirror2_i. The HIP integer VIF kernel used clamp boundary mode instead of mirror2 (the CPU reference), causing parity failures at the frame boundary for non-trivial content. Supersedes the ADR-0566 claim that clamp matches CPU within places=4. | Accepted | 2026-06-13 | hip, vif, parity, boundary, correctness, fork-local |
| [ADR-1104](1104-float-vif-avx512-golden-regression-fix.md) | Remove AVX-512 dispatch from float VIF convolution to restore Netflix golden scores. ADR-0504 added AVX-512 float convolution dispatch but the wider FMA partial-sum tree produces different rounding than AVX2, causing the Netflix golden VMAFEXEC_score assertion (76.66740433333332, places=4) to fail on AVX-512 CPUs. Fix: remove HAVE_AVX512 dispatch blocks from vif_filter1d_s/sq_s/xy_s; float VIF uses AVX2 path matching upstream Netflix/vmaf. All 271 golden tests pass after fix. | Accepted | 2026-06-13 | simd, correctness, float-vif, bug-fix, fork-local |
| [ADR-1105](1105-ensemble-v2-prod-flip-deferred-oneshot-retrain.md) | Defer the `fr_regressor_v2_ensemble` production flip (ADR-0321) to the locked one-shot post-RC retrain. The ADR-0321 production weights were LOSO-validated at codec_vocab=14; the vocab was trimmed to 6, so #865 regenerated the ONNX in smoke mode (smoke=true) to keep the load path correct. For RC: restore the license/license_url/sigstore_bundle fields #865 dropped, keep smoke=true (matches shipped weights), and mark `test_fr_regressor_v2_ensemble_seed_rows_are_production` xfail(strict=True) so it auto-fails the moment the one-shot retrain lands real weights. No Netflix golden assertion touched. | Accepted | 2026-06-13 | ai, models, rc, docs, fork-local |
40 changes: 25 additions & 15 deletions docs/ai/models/fr_regressor_v2_probabilistic.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,27 +8,37 @@ form _"give me the CRF where the **lower** bound of the 95 % interval is
still ≥ 92"_ — driving the new `vmaf-tune --quality-confidence` flag
(planned, see [ADR-0237](../../adr/0237-quality-aware-encode-automation.md)).

> **Status — production (per-seed `smoke: false`).** As of 2026-05-06
> the five `fr_regressor_v2_ensemble_v1_seed{0..4}` rows in
> `model/tiny/registry.json` carry **production** ONNX weights:
> trained on the 5,640-row Phase A canonical-6 corpus (9 Netflix
> sources × `h264_nvenc`) after the 9-fold LOSO ship-gate cleared
> with mean PLCC 0.997 (spread 0.001) per
> `runs/ensemble_v2_real/PROMOTE.json`. Each non-smoke ONNX has a
> matching sidecar `fr_regressor_v2_ensemble_v1_seed{N}.json` with
> the canonical encoder-vocab-v2 + codec-block layout + training
> recipe + per-seed gate evidence. Fresh seed exports also include ADR-0661
> `run_provenance` with the corpus, PROMOTE verdict, argv, output seed
> sidecars, and optional registry target. The shared
> `fr_regressor_v2_ensemble_v1.json` manifest is unchanged. See
> **Status — smoke placeholder for the RC; production flip deferred to
> the one-shot post-RC retrain ([ADR-1105](../../adr/1105-ensemble-v2-prod-flip-deferred-oneshot-retrain.md)).**
> The five `fr_regressor_v2_ensemble_v1_seed{0..4}` rows in
> `model/tiny/registry.json` currently carry `smoke: true`. The
> ADR-0321 production flip (2026-05-06) shipped LOSO-validated weights
> (mean PLCC 0.997, spread 0.001 per `runs/ensemble_v2_real/PROMOTE.json`)
> trained against a codec one-hot of width 14. `codec_vocab` was later
> trimmed to 6, which made those weights' input dimension stale; PR #865
> regenerated the on-disk ONNX at the correct width but in `--smoke`
> mode (1 epoch, synthetic corpus) to keep the load path working. Those
> smoke weights are placeholders, not a production fit. Re-establishing
> production at `codec_vocab=6` requires re-running
> `export_ensemble_v2_seeds.py`, which is part of the locked one-shot
> post-RC retrain (the ensemble is in scope). Until then,
> `test_fr_regressor_v2_ensemble_seed_rows_are_production` is marked
> `xfail(strict=True)`; it auto-fails the suite the moment the retrain
> lands real weights (`smoke: false` + matching sidecar sha), forcing
> removal of the marker. The per-seed sidecars
> (`fr_regressor_v2_ensemble_v1_seed{N}.json`) still describe the older
> production weights and retain their PROMOTE provenance; the one-shot
> retrain regenerates ONNX + sidecars together. See
> [ADR-0303](../../adr/0303-fr-regressor-v2-ensemble-prod-flip.md)
> (gate definition),
> [ADR-0309](../../adr/0309-fr-regressor-v2-ensemble-real-corpus-retrain.md)
> (separate-PR rule),
> [ADR-0319](../../adr/0319-ensemble-loso-trainer-real-impl.md) (LOSO
> trainer), and
> trainer),
> [ADR-0321](../../adr/0321-fr-regressor-v2-ensemble-full-prod-flip.md)
> (this flip). The scaffold-era ADR-0279 entry point is preserved
> (the original production flip), and
> [ADR-1105](../../adr/1105-ensemble-v2-prod-flip-deferred-oneshot-retrain.md)
> (RC deferral). The scaffold-era ADR-0279 entry point is preserved
> for history.

## What the output means
Expand Down
1 change: 1 addition & 0 deletions docs/state.md
Original file line number Diff line number Diff line change
Expand Up @@ -296,6 +296,7 @@ landed fix yet._
<!-- T-CUDA-MOTION-SAD-BATCH-PENDING-2026-05-29 moved to Recently Closed — PR #217 merged 2026-06-03 (ADR-0845) -->
| ~~**T-SYCL-CLANG-TIDY-DISABLED**~~ — `clang-tidy-sycl` CI job re-enabled 2026-06-08 via `scripts/ci/gen-sycl-compile-commands.py` (synthesises compile_commands.json entries for the meson CUSTOM_COMMAND SYCL TUs) + the existing `scripts/ci/clang-tidy-sycl.sh` wrapper (`-D__SYCL_DEVICE_ONLY__=0` guards `__spirv_ControlBarrier`/`__ocl_event_t`). Job stays `continue-on-error: true` (advisory) until one green master run confirms the approach holds. ([ADR-0623](adr/0623-scaffold-audit-p2-half-finished.md)) | `grep "clang-tidy-sycl:" .github/workflows/lint-and-format.yml` — job present without `if: false`. | Advisory. | Closes (promoted to required gate) after one green master run and re-addition to the Required Checks list. |
<!-- T-DOCKER-SMOKE moved to Recently closed — promoted to blocking 2026-06-08, chore/promote-docker-smoke-blocking -->
| **T-ENSEMBLE-V2-PROD-FLIP-DEFERRED-2026-06-13** | The five `fr_regressor_v2_ensemble_v1_seed{0..4}` rows ship at smoke quality for the RC. ADR-0321 promoted them to production with LOSO-validated weights trained at `codec_vocab=14`; the vocab was trimmed to 6, so PR #865 regenerated the ONNX in `--smoke` mode (1 epoch, synthetic) to keep the load path correct and set `smoke: true`. PR #865 also dropped `license`/`license_url`/`sigstore_bundle` from the five rows — restored here. The production flip is deferred to the locked one-shot post-RC retrain (ensemble is in scope), per [ADR-1105](adr/1105-ensemble-v2-prod-flip-deferred-oneshot-retrain.md). `test_fr_regressor_v2_ensemble_seed_rows_are_production` is `xfail(strict=True)` so it auto-fails the moment real weights land. | `python3 -m pytest python/test/model_registry_schema_test.py -q` → 10 passed, 1 xfailed (the deferred production assertion). | One-shot retrain (post-RC, locked plan). | Closes when the one-shot retrain re-runs `export_ensemble_v2_seeds.py` at `codec_vocab=6`, flips `smoke: false`, and removes the xfail marker (test xpasses → strict failure forces marker removal). |

## Deferred (waiting on external dataset access)

Expand Down
Loading
Loading