Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .clang-tidy
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ WarningsAsErrors: >
bugprone-use-after-move,
bugprone-integer-division,
bugprone-infinite-loop,
cert-err33-c,
cert-flp30-c,
cert-msc30-c,
cert-msc50-cpp,
Expand Down
21 changes: 10 additions & 11 deletions .github/workflows/libvmaf-build-matrix.yml
Original file line number Diff line number Diff line change
Expand Up @@ -128,19 +128,18 @@ jobs:
meson_extra: -Denable_hip=true
name: Ubuntu HIP

# --- Metal T8-1 scaffold (ADR-0361) ---
# Stub-only build — every Metal entry point returns -ENOSYS
# until the runtime PR (T8-1b) lands. macos-latest ships the
# Metal SDK as part of the system framework set
# (`/System/Library/Frameworks/Metal.framework` is guaranteed
# to exist), so the scaffold's
# --- Metal backend (ADR-0361 scaffold, ADR-0420 runtime) ---
# Builds the Metal runtime and every `.metal` kernel with
# `-Denable_metal=enabled`. macos-latest ships the Metal SDK as part
# of the system framework set (`/System/Library/Frameworks/
# Metal.framework` is guaranteed to exist), so the
# `dependency('Metal', required: false)` /
# `dependency('MetalKit', required: false)` probes resolve
# cleanly. The smoke test runs (no real GPU work needed —
# every assertion checks the -ENOSYS / -EINVAL contract).
# `dependency('MetalKit', required: false)` probes resolve. A hosted
# runner has no GPU worth scoring on: the tests that need a device
# skip, the rest check the -ENOSYS / -EINVAL / -ENODEV contracts.
# Apple Silicon (M1+ / GPU Family Apple 7+) is the runtime
# target; the scaffold lane builds on whichever macos-latest
# arch GitHub provides today (Apple Silicon as of 2026).
# target; the lane builds on whichever macos-latest arch GitHub
# provides today (Apple Silicon as of 2026).
# required-aggregator
- os: macos-latest
CC: ccache clang
Expand Down
32 changes: 32 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -1193,6 +1193,13 @@ make `core/AGENTS.md` a generated index over `AGENTS.d/` topic pages ([ADR-1454]
off by default; the documented cache flags never existed).


- `make docs-build` runs the strict MkDocs build the docs workflow and the
pre-push gate run, and `make docs-serve` starts the live preview. The Metal
lane comment of the build matrix no longer calls the build stub-only, ADR-0581
names ADR-0597 in its status line, and the rebase notes carry the
`vmaf_cuda_picture_get_pix_fmt()` accessor of PR #1118.


- The CUDA / NVDEC path of the `libvmaf_cuda` FFmpeg filter is no longer called
"zero-copy" in the documentation: no frame goes through host memory, but each
decoded frame is copied device to device into libvmaf's picture pool
Expand Down Expand Up @@ -2751,6 +2758,12 @@ make `core/AGENTS.md` a generated index over `AGENTS.d/` topic pages ([ADR-1454]
`docs/development/release-workflow-verification.md`.


- `cert-err33-c` (an ignored return value of a standard-library call) is in the
`WarningsAsErrors` list of `.clang-tidy`. Every clang-tidy lane already measured
zero findings of it, so the promotion ADR-0694 asked for changes no count; a new
unchecked `fclose()` or `fputs()` now fails `Tidy Changed`.


- `scripts/ci/check-tidy-coverage.py` (pre-commit hook `check-tidy-coverage`) fails when a tracked C, C++, CUDA, HIP, Objective-C++ or Metal translation unit is in no clang-tidy lane's measured sources and not in `.config/lint-exceptions.d/clang-tidy-coverage.toml`; the Metal kernels, the Pelorus mirror and the other files no tool can read are listed there with a reason and an expiry. `write-compile-commands.py` now exports the Objective-C and Objective-C++ rules, so the macOS lane reads the `.mm` files. Details: [tidy lanes](docs/development/tidy-lanes.md). The macOS lane's baseline (`scripts/ci/tidy-baseline-metal.json`) records 939 findings in the Metal host code; `Tidy Metal` takes a `fix` dispatch input that uploads clang-tidy's own fixes as a patch.


Expand Down Expand Up @@ -3057,6 +3070,13 @@ make `core/AGENTS.md` a generated index over `AGENTS.d/` topic pages ([ADR-1454]
The two model cards describe what the generators write: the ONNX file only.


- The four FR regressor trainers (`train_fr_regressor.py`, `_v2.py`,
`_v2_ensemble.py`, `_v3.py`) write `model/tiny/registry.json` through
`vmaf_train.registry.write_registry_json()`. A non-finite number in a registry
row is written as `null` instead of the non-standard `NaN` token no strict JSON
reader accepts; a registry of finite values is byte-identical to before.


- **`libvmaf.h` and the API guide say what an index gap and an early query do.**
`vmaf_read_pictures()` has always rejected a repeated or earlier index with
`-EINVAL`; it also accepts an index that skips values, and then the motion
Expand Down Expand Up @@ -3738,6 +3758,18 @@ make `core/AGENTS.md` a generated index over `AGENTS.d/` topic pages ([ADR-1454]
`test_cython_adm_dwt_band_decl_contract` keeps them in step with `adm.c`.


- **macOS release builds no longer branch from `close()` into a feature
extractor.** macOS declares the C library's `close()` with an assembler
label, and in a full-LTO link (the release default) that symbol and the
CPU extractors' `static close()` callbacks became the same symbol: the
`close()` on the fdopen() failure paths of the output file, the CAMBI
heatmap file and the SVM model save called an extractor's close with a
file descriptor and crashed, and `test_adm_coverage` crashed on every macOS
leg. The callbacks are named `close_fex`, and
`test_libc_named_internal_functions` refuses a static C function named
after a C library function.


- The `vmaf-dev-mcp` container starts again on an image built from the
current `dev/Containerfile`. Its entrypoint ran `chmod 1777 /tmp` as the
unprivileged `vmaf` user; uutils coreutils 0.10 in the updated Ubuntu
Expand Down
12 changes: 12 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -902,6 +902,18 @@ hiss-coverage:
dedupe-check:
@standardsctl dedupe scan .

# Documentation site (mkdocs.yml writes build-docs/site). docs-build is the
# strict build the docs workflow and the pre-push MkDocs gate run; docs-serve
# is the live preview while editing under docs/.
.PHONY: docs-build docs-serve
docs-build:
@command -v mkdocs >/dev/null || { echo "mkdocs not found: pip install -r docs/requirements.txt"; exit 1; }
mkdocs build --strict

docs-serve:
@command -v mkdocs >/dev/null || { echo "mkdocs not found: pip install -r docs/requirements.txt"; exit 1; }
mkdocs serve

# BEGIN praetor documentation gate
.PHONY: docs-lint docs-figures
verify-all: docs-lint docs-figures
Expand Down
3 changes: 2 additions & 1 deletion ai/scripts/train_fr_regressor.py
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@

from aiutils.file_utils import sha256
from aiutils.run_manifest import build_run_provenance, write_manifest_json
from vmaf_train.registry import write_registry_json

REPO_ROOT = Path(__file__).resolve().parents[2]
SCRIPT_PATH = Path(__file__).resolve()
Expand Down Expand Up @@ -308,7 +309,7 @@ def _upsert_registry_entry(registry_path: Path, onnx_path: Path, digest: str, no
models.append(new_entry)
models.sort(key=lambda e: e.get("id", ""))
registry["models"] = models
registry_path.write_text(json.dumps(registry, indent=2, sort_keys=True) + "\n")
write_registry_json(registry_path, registry)


def _export_and_register(
Expand Down
3 changes: 2 additions & 1 deletion ai/scripts/train_fr_regressor_v2.py
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,7 @@

from aiutils.file_utils import sha256
from aiutils.run_manifest import build_run_provenance, write_manifest_json
from vmaf_train.registry import write_registry_json

REPO_ROOT = Path(__file__).resolve().parents[2]
SCRIPT_PATH = Path(__file__).resolve()
Expand Down Expand Up @@ -624,7 +625,7 @@ def _update_fr_v2_registry(
models.append(new_entry)
models.sort(key=lambda e: e.get("id", ""))
registry["models"] = models
registry_path.write_text(json.dumps(registry, indent=2, sort_keys=True) + "\n")
write_registry_json(registry_path, registry)


def _write_sidecar_and_registry(
Expand Down
3 changes: 2 additions & 1 deletion ai/scripts/train_fr_regressor_v2_ensemble.py
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,7 @@

from aiutils.file_utils import sha256 # noqa: E402
from aiutils.run_manifest import build_run_provenance, write_manifest_json # noqa: E402
from vmaf_train.registry import write_registry_json # noqa: E402

CANONICAL_6: tuple[str, ...] = (
"adm2",
Expand Down Expand Up @@ -356,7 +357,7 @@ def _update_registry(
)
keep.sort(key=lambda e: e.get("id", ""))
registry["models"] = keep
registry_path.write_text(json.dumps(registry, indent=2, sort_keys=True) + "\n")
write_registry_json(registry_path, registry)


def _build_parser() -> argparse.ArgumentParser:
Expand Down
3 changes: 2 additions & 1 deletion ai/scripts/train_fr_regressor_v3.py
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@

from aiutils.file_utils import sha256
from aiutils.run_manifest import build_run_provenance, write_manifest_json
from vmaf_train.registry import write_registry_json

REPO_ROOT = Path(__file__).resolve().parents[2]
SCRIPT_PATH = Path(__file__).resolve()
Expand Down Expand Up @@ -669,7 +670,7 @@ def _upsert_v3_registry_row(registry_path: Path, row: dict[str, Any]) -> None:
models.append(row)
models.sort(key=lambda e: e.get("id", ""))
registry["models"] = models
registry_path.write_text(json.dumps(registry, indent=2, sort_keys=True) + "\n")
write_registry_json(registry_path, registry)


def write_sidecar_and_registry(
Expand Down
10 changes: 5 additions & 5 deletions ai/src/vmaf_train/registry.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
VALID_KINDS = {"fr", "nr", "filter"}


class ModelMetadata(BaseModel):
class ModelMetadata(BaseModel): # type: ignore[misc] # pydantic is untyped for the mypy gate (--no-site-packages)
"""Sidecar JSON describing a shipped tiny model.

Migrated from ``@dataclass`` to ``pydantic.BaseModel`` so the load
Expand Down Expand Up @@ -59,7 +59,7 @@ class ModelMetadata(BaseModel):
cosign_signature: str | None = None
notes: str | None = None

@field_validator("kind")
@field_validator("kind") # type: ignore[untyped-decorator] # pydantic is untyped for the mypy gate (--no-site-packages)
@classmethod
def _valid_kind(cls, v: str) -> str:
if v not in VALID_KINDS:
Expand Down Expand Up @@ -138,7 +138,7 @@ def register(

def load(sidecar_path: Path) -> ModelMetadata:
doc: dict[str, Any] = json.loads(sidecar_path.read_text())
return ModelMetadata.model_validate(doc)
return ModelMetadata.model_validate(doc) # type: ignore[no-any-return] # pydantic is untyped for the mypy gate (--no-site-packages)


def _sanitize_nonfinite(obj: Any) -> Any:
Expand Down Expand Up @@ -167,7 +167,7 @@ def _sanitize_nonfinite(obj: Any) -> Any:
return holder[0]


def dumps_registry_json(payload: dict, **kwargs: Any) -> str:
def dumps_registry_json(payload: dict[str, Any], **kwargs: Any) -> str:
"""Serialise a registry payload to a pretty-printed, non-finite-safe JSON string.

NaN and Infinity values inside *payload* are replaced with ``null`` so the
Expand All @@ -187,7 +187,7 @@ def dumps_registry_json(payload: dict, **kwargs: Any) -> str:
return json.dumps(_sanitize_nonfinite(payload), **kwargs)


def write_registry_json(path: Path, payload: dict, **kwargs: Any) -> None:
def write_registry_json(path: Path, payload: dict[str, Any], **kwargs: Any) -> None:
"""Write *payload* as pretty-printed, newline-terminated JSON to *path*.

Convenience wrapper around :func:`dumps_registry_json` that appends a
Expand Down
88 changes: 88 additions & 0 deletions ai/tests/test_training_scripts_registry_nonfinite.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
# Copyright 2026 Lusoris
# SPDX-License-Identifier: EUPL-1.2
"""The FR regressor trainers write ``model/tiny/registry.json`` as strict JSON.

``T-BUG048-AI-SCRIPT-HELPERS-2026-09-26`` (registry half): the four trainers
wrote the registry with a raw ``json.dumps``, which prints ``NaN`` for a
non-finite value and produces a file no strict reader accepts. They now go
through :func:`vmaf_train.registry.write_registry_json`. Positive: a registry
row that already holds a non-finite number comes back as ``null``. Negative: no
``NaN`` / ``Infinity`` token survives. Boundary: a registry of finite values is
byte-identical to the old ``json.dumps(..., indent=2, sort_keys=True)`` output.
"""

from __future__ import annotations

import json
import sys
from collections.abc import Callable
from pathlib import Path

REPO_ROOT = Path(__file__).resolve().parents[2]
sys.path.insert(0, str(REPO_ROOT / "ai" / "scripts"))
sys.path.insert(0, str(REPO_ROOT / "ai" / "src"))

# pylint: disable=wrong-import-position
import train_fr_regressor as v1 # noqa: E402
import train_fr_regressor_v2 as v2 # noqa: E402
import train_fr_regressor_v2_ensemble as ensemble # noqa: E402
import train_fr_regressor_v3 as v3 # noqa: E402

_SEED_ROW = '{"id": "seed_row", "kind": "fr", "onnx": "seed.onnx", "plcc": %s}'


def _reject(token: str) -> object:
raise AssertionError(f"registry.json holds the non-standard token {token}")


def _upsert_v1(path: Path) -> None:
v1._upsert_registry_entry(path, Path("fr_regressor_v1.onnx"), "0" * 64, "notes")


def _upsert_v2(path: Path) -> None:
v2._update_fr_v2_registry(path, "fr_regressor_v2.onnx", "0" * 64, "notes", True)


def _upsert_ensemble(path: Path) -> None:
member = {"id": "ens_seed0", "onnx": "ens_seed0.onnx", "sha256": "0" * 64, "seed": 0}
ensemble._update_registry(path, ensemble_id="ens", members=[member], smoke=True)


def _upsert_v3(path: Path) -> None:
v3._upsert_v3_registry_row(path, {"id": "fr_regressor_v3", "kind": "fr"})


_UPSERTS: dict[str, Callable[[Path], None]] = {
"v1": _upsert_v1,
"v2": _upsert_v2,
"ensemble": _upsert_ensemble,
"v3": _upsert_v3,
}


def test_registry_holding_a_non_finite_value_is_rewritten_as_strict_json(
tmp_path: Path,
) -> None:
for name, upsert in _UPSERTS.items():
for token in ("NaN", "Infinity", "-Infinity"):
registry = tmp_path / f"{name}-{token}.json"
registry.write_text('{"models": [' + _SEED_ROW % token + "]}\n", encoding="utf-8")

upsert(registry)

text = registry.read_text(encoding="utf-8")
rows = json.loads(text, parse_constant=_reject)["models"]
seed = next(row for row in rows if row["id"] == "seed_row")
assert seed["plcc"] is None, (name, token)
assert text.endswith("\n"), (name, token)


def test_finite_registry_keeps_the_previous_byte_layout(tmp_path: Path) -> None:
for name, upsert in _UPSERTS.items():
registry = tmp_path / f"{name}.json"
registry.write_text('{"models": [' + _SEED_ROW % "0.5" + "]}\n", encoding="utf-8")

upsert(registry)

text = registry.read_text(encoding="utf-8")
assert text == json.dumps(json.loads(text), indent=2, sort_keys=True) + "\n", name
5 changes: 5 additions & 0 deletions changelog.d/changed/docs-build-make-targets.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
- `make docs-build` runs the strict MkDocs build the docs workflow and the
pre-push gate run, and `make docs-serve` starts the live preview. The Metal
lane comment of the build matrix no longer calls the build stub-only, ADR-0581
names ADR-0597 in its status line, and the rebase notes carry the
`vmaf_cuda_picture_get_pix_fmt()` accessor of PR #1118.
4 changes: 4 additions & 0 deletions changelog.d/changed/tidy-cert-err33-errors.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
- `cert-err33-c` (an ignored return value of a standard-library call) is in the
`WarningsAsErrors` list of `.clang-tidy`. Every clang-tidy lane already measured
zero findings of it, so the promotion ADR-0694 asked for changes no count; a new
unchecked `fclose()` or `fputs()` now fails `Tidy Changed`.
5 changes: 5 additions & 0 deletions changelog.d/fixed/ai-training-registry-strict-json.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
- The four FR regressor trainers (`train_fr_regressor.py`, `_v2.py`,
`_v2_ensemble.py`, `_v3.py`) write `model/tiny/registry.json` through
`vmaf_train.registry.write_registry_json()`. A non-finite number in a registry
row is written as `null` instead of the non-standard `NaN` token no strict JSON
reader accepts; a registry of finite values is byte-identical to before.
10 changes: 10 additions & 0 deletions changelog.d/fixed/darwin-lto-static-close.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
- **macOS release builds no longer branch from `close()` into a feature
extractor.** macOS declares the C library's `close()` with an assembler
label, and in a full-LTO link (the release default) that symbol and the
CPU extractors' `static close()` callbacks became the same symbol: the
`close()` on the fdopen() failure paths of the output file, the CAMBI
heatmap file and the SVM model save called an extractor's close with a
file descriptor and crashed, and `test_adm_coverage` crashed on every macOS
leg. The callbacks are named `close_fex`, and
`test_libc_named_internal_functions` refuses a static C function named
after a C library function.
42 changes: 42 additions & 0 deletions core/src/feature/AGENTS.d/libc-named-statics.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
---
paths:
- core/src/feature/brisque.c
- core/src/feature/ciede.c
- core/src/feature/delta_e_itp.c
- core/src/feature/float_adm.c
- core/src/feature/float_moment.c
- core/src/feature/float_motion.c
- core/src/feature/float_ms_ssim.c
- core/src/feature/float_psnr.c
- core/src/feature/float_ssim.c
- core/src/feature/float_vif.c
- core/src/feature/integer_adm.c
- core/src/feature/integer_ssim.c
- core/src/feature/integer_vif.c
- core/src/feature/niqe.c
- core/src/feature/pu21.c
- core/src/feature/speed.c
- core/src/feature/ssimulacra2.c
- core/test/test_libc_named_internal_functions.py
invariant: no static C function named after a C library function; extractor close callback is close_fex.
---
<!-- markdownlint-disable MD013 MD032 MD060 -->
# Extractor callbacks never take a C library name

## `close_fex`, not `close` (T-DARWIN-LTO-STATIC-CLOSE-COLLISION-2026-10-06)

Upstream Netflix/vmaf names every CPU extractor's close callback `static int
close(VmafFeatureExtractor *)`. macOS headers declare `close()` with an
assembler label (`__DARWIN_ALIAS_C(close)` = `__asm("_close")`). Full LTO
(`b_lto=true`, the release default) merges all modules: the label keeps the
declaration's IR name apart from the static one, so the IR linker does not
rename the static, and both emit as `_close`. Every libc `close()` call in the
merged module then branches into an extractor's close with a file descriptor
for `fex` (macOS `test_adm_coverage` SIGSEGV; the fdopen() failure paths of
`libvmaf.c`, `cambi.c`, `svm.cpp`). Linux glibc has no label: no collision.

The fork names the callback `close_fex` (the GPU twins already use
`close_fex_cuda` / `_hip` / `_metal`). An upstream sync that touches one of
these files keeps the fork's name; a new extractor or helper never defines a
`static` function named after a C library or POSIX function.
`core/test/test_libc_named_internal_functions.py` (fast suite) fails on one.
1 change: 1 addition & 0 deletions core/src/feature/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ PSNR, SSIM, MS-SSIM, LPIPS, …). Parent: [../../AGENTS.md](../../AGENTS.md).
| `iqa/convolve.c`, `iqa/ssim_tools.c` | [iqa](AGENTS.d/iqa.md) | IQA/VIF SIMD helper decomposition and reserved-identifier cleanup invariants. |
| `/core/test/test_feature_isa_invariance.c` | [isa-invariance](AGENTS.d/isa-invariance.md) | Feature scores must not depend on host ISA; promotions and reductions must match scalar. |
| `vif_tools.c`, `vif_tools.h` | [lanczos4](AGENTS.d/lanczos4.md) | lanczos4 prescale weights single implementation shared with GPU twins. |
| `brisque.c`, `ciede.c`, `delta_e_itp.c`, `float_adm.c`, `float_moment.c`, `float_motion.c`, `float_ms_ssim.c`, `float_psnr.c`, `float_ssim.c`, `float_vif.c`, `integer_adm.c`, `integer_ssim.c`, `integer_vif.c`, `niqe.c`, `pu21.c`, `speed.c`, `ssimulacra2.c`, `/core/test/test_libc_named_internal_functions.py` | [libc-named-statics](AGENTS.d/libc-named-statics.md) | no static C function named after a C library function; extractor close callback is close_fex. |
| `float_motion.c`, `float_vif.c` | [min-dimension-guards](AGENTS.d/min-dimension-guards.md) | Minimum-dimension guards cover every plane, not just luma. |
| `feature_mobilesal.c`, `/core/test/test_mobilesal.c`, `/core/test/dnn/test_mobilesal_run.c` | [mobilesal](AGENTS.d/mobilesal.md) | MobileSal extractor pads sides to a multiple of 8 for the students and averages only the frame's own area. |
| `feature_collector.cpp`, `feature_name.cpp` | [model-options](AGENTS.d/model-options.md) | Model options gate GPU twin selection; every option must be parsed or safely rejected. |
Expand Down
Loading
Loading