chore(ci): audit per-file coverage overrides — tighten tiny_extractor_template.h 10 → 75 by lusoris · Pull Request #370 · VMAFx/vmafx

lusoris · 2026-05-30T19:36:25Z

Summary

Periodic audit of scripts/ci/coverage-check.sh's PER_FILE_MIN map (ADR-0114). Cross-references each entry against fresh measured coverage; identifies new files that should have overrides; codifies the audit rule into a recurring policy (ADR-0881).

Per-override action table

File	Override pre-audit	Actual	Action	Rationale
`core/src/dnn/ort_backend.c`	78 %	77.8 %	Keep at 78	At cap. PR #338 in flight adds `vmaf_ort_output_name_at` test (77.8 % → 78.5 %) without raising the bar. Correct pattern per ADR-0114.
`core/src/dnn/dnn_api.c`	78 %	78.0 %	Keep at 78	At cap. Structural ceiling rationale per ADR-0114 §Context unchanged (EP-attach error paths + dead `has_norm` branch).
`core/src/dnn/tiny_extractor_template.h`	10 %	77.4 %	Tighten 10 → 75	67.4 pp slack — silently giving up regression-lock. Original 10 % was set when only 1 extractor instantiated the helpers; 4 do today (`feature_lpips`, `fastdvdnet_pre`, `feature_mobilesal`, `feature_transnet_v2`). 75 % = 2.4 pp slack, mirroring ADR-0114's risk tolerance.

New files surveyed (none flagged)

All other dnn/ files plus opt.c and read_json_model.c clear the global 85 % critical floor — no new override entries required:

File	Coverage
`core/src/dnn/dnn_attach_api.c`	92.0 %
`core/src/dnn/model_loader.c`	86.9 %
`core/src/dnn/onnx_scan.c`	93.4 %
`core/src/dnn/op_allowlist.c`	100.0 %
`core/src/dnn/tensor_io.c`	98.1 %
`core/src/opt.c`	100.0 %
`core/src/read_json_model.c`	88.0 %

Codified audit rule (lands in ADR-0881)

Tighten when slack > 5 pp — ratchet to actual − 2 pp.
Keep at-cap entries (slack ≤ 2 pp).
Remove when actual ≥ global 85 % floor.
Audit cadence: quarterly + before any PER_FILE_MIN edit.

Policy guard

Per CLAUDE.md "never lower a threshold to bypass" — this PR only tightens or keeps. PR #338 (ort_backend.c) is the example of the correct "add tests, do not lower the bar" pattern.

Reproducer

PKG_CONFIG_PATH=/path/to/onnxruntime/lib/pkgconfig \
  meson setup core/build-coverage core --buildtype=debug \
    -Db_coverage=true -Denable_cuda=false -Denable_sycl=false \
    -Denable_float=true -Denable_avx512=true -Denable_dnn=enabled \
    -Dc_args=-fprofile-update=atomic -Dcpp_args=-fprofile-update=atomic
ninja -C core/build-coverage
LD_LIBRARY_PATH=/path/to/onnxruntime/lib \
  meson test -C core/build-coverage --num-processes 1
cd core && gcovr --root .. --filter 'src/.*' \
  --exclude '.*/test/.*' --exclude '.*/tests/.*' \
  --json-summary build-coverage/coverage.json build-coverage
bash scripts/ci/coverage-check.sh core/build-coverage/coverage.json 37 85
# Verify:
#   tiny_extractor_template.h now reports "min 75%" (was "min 10%")
#   tiny_extractor_template.h 77.4% > 75% → PASS
#   ort_backend.c 77.8% < 78% → FAIL (PR #338 territory — not this PR's scope)

Type

chore — CI gate threshold tightening (no production-code behaviour change)

Bug-status hygiene (ADR-0165)

no state delta: coverage-overrides audit is a CI process/hygiene change — not a tracked bug open/close/not-affected event per ADR-0165.

Netflix golden-data gate (ADR-0024)

No Netflix assertAlmostEqual(...) scores touched.

Deep-dive deliverables (ADR-0108)

Research digest — docs/research/0881-coverage-overrides-audit-2026-05-30.md with per-file CSV, reusable audit procedure, before/after gate output.
Decision matrix — ADR-0881 ## Alternatives considered covers 5 options (tighten-77 / tighten-70 / remove-entry / add-fault-injection-tests / leave-at-10).
AGENTS.md invariant note — scripts/ci/AGENTS.md rebase-sensitive surfaces table now lists coverage-check.sh with the PER_FILE_MIN ADR-citation invariant + gcovr path-format coupling.
Reproducer / smoke-test command — see Reproducer block above.
CHANGELOG fragment — changelog.d/changed/0881-coverage-overrides-audit-2026-05-30.md added.
Rebase note — docs/rebase-notes.md § "Coverage-overrides audit — tighten tiny_extractor_template.h (ADR-0881, 2026-05-30)" with no rebase impact: REASON (fork-only files only).

🤖 Generated with Claude Code

…_template.h 10 → 75 Periodic audit of `scripts/ci/coverage-check.sh`'s `PER_FILE_MIN` map (ADR-0114). Three findings: 1. `core/src/dnn/tiny_extractor_template.h` — actual 77.4 %, override 10 % (67.4 pp slack). Original 10 % cap was set when only one extractor (`feature_lpips.c`) instantiated the inline helpers; four extractors now do (`feature_lpips`, `fastdvdnet_pre`, `feature_mobilesal`, `feature_transnet_v2`). Tighten to 75 (2.4 pp slack, mirroring the 1.3-1.7 pp slack ADR-0114 used for the at-cap entries). Locks 65 pp of de-facto regression-coverage. 2. `core/src/dnn/ort_backend.c` — actual 77.8 %, override 78 % (at cap, currently failing on master). Keep at 78 — PR #338 in flight adds `vmaf_ort_output_name_at` unit test, lifting actual to 78.5 % without raising the bar (correct pattern per ADR-0114). 3. `core/src/dnn/dnn_api.c` — actual 78.0 %, override 78 % (at cap). Keep at 78 — structural ceiling rationale per ADR-0114 §Context unchanged. No new override entries required. All other dnn/ files plus opt.c and read_json_model.c clear the global 85 % critical floor (dnn_attach_api.c 92 %, model_loader.c 87 %, onnx_scan.c 93 %, op_allowlist.c 100 %, tensor_io.c 98 %, opt.c 100 %, read_json_model.c 88 %). ADR-0881 also codifies the recurring audit rule (tighten when slack > 5 pp; keep at-cap; remove when actual ≥ global 85 % floor; audit quarterly + before any PER_FILE_MIN edit). Audit procedure documented in the companion research digest. Reproducer: PKG_CONFIG_PATH=/path/to/onnxruntime/lib/pkgconfig \ meson setup core/build-coverage core --buildtype=debug \ -Db_coverage=true -Denable_cuda=false -Denable_sycl=false \ -Denable_float=true -Denable_avx512=true -Denable_dnn=enabled \ -Dc_args=-fprofile-update=atomic -Dcpp_args=-fprofile-update=atomic ninja -C core/build-coverage LD_LIBRARY_PATH=/path/to/onnxruntime/lib \ meson test -C core/build-coverage --num-processes 1 cd core && gcovr --root .. --filter 'src/.*' \ --exclude '.*/test/.*' --exclude '.*/tests/.*' \ --json-summary build-coverage/coverage.json build-coverage bash scripts/ci/coverage-check.sh core/build-coverage/coverage.json 37 85 # Pre-audit: tiny_extractor_template.h reports "min 10%" # Post-audit: same file reports "min 75%" — gate enforces the new floor Closes the ADR-0114 implicit follow-up "re-audit slack overrides on a cadence so they don't rot". Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

lusoris · 2026-05-31T13:15:03Z

Superseded by master after the 117-PR merge marathon of 2026-05-31. Diff-extract produced empty rebase (AHEAD=0), indicating intended content already landed via sibling merges.

lusoris · 2026-06-01T19:46:04Z

Superseded by #528 — bundled per 2026-06-01 triage.

…test + #446 SYCL parity round 3) (#528) * chore(ci): audit per-file coverage overrides — tighten tiny_extractor_template.h 10 → 75 Periodic audit of `scripts/ci/coverage-check.sh`'s `PER_FILE_MIN` map (ADR-0114). Three findings: 1. `core/src/dnn/tiny_extractor_template.h` — actual 77.4 %, override 10 % (67.4 pp slack). Original 10 % cap was set when only one extractor (`feature_lpips.c`) instantiated the inline helpers; four extractors now do (`feature_lpips`, `fastdvdnet_pre`, `feature_mobilesal`, `feature_transnet_v2`). Tighten to 75 (2.4 pp slack, mirroring the 1.3-1.7 pp slack ADR-0114 used for the at-cap entries). Locks 65 pp of de-facto regression-coverage. 2. `core/src/dnn/ort_backend.c` — actual 77.8 %, override 78 % (at cap, currently failing on master). Keep at 78 — PR #338 in flight adds `vmaf_ort_output_name_at` unit test, lifting actual to 78.5 % without raising the bar (correct pattern per ADR-0114). 3. `core/src/dnn/dnn_api.c` — actual 78.0 %, override 78 % (at cap). Keep at 78 — structural ceiling rationale per ADR-0114 §Context unchanged. No new override entries required. All other dnn/ files plus opt.c and read_json_model.c clear the global 85 % critical floor (dnn_attach_api.c 92 %, model_loader.c 87 %, onnx_scan.c 93 %, op_allowlist.c 100 %, tensor_io.c 98 %, opt.c 100 %, read_json_model.c 88 %). ADR-0881 also codifies the recurring audit rule (tighten when slack > 5 pp; keep at-cap; remove when actual ≥ global 85 % floor; audit quarterly + before any PER_FILE_MIN edit). Audit procedure documented in the companion research digest. Reproducer: PKG_CONFIG_PATH=/path/to/onnxruntime/lib/pkgconfig \ meson setup core/build-coverage core --buildtype=debug \ -Db_coverage=true -Denable_cuda=false -Denable_sycl=false \ -Denable_float=true -Denable_avx512=true -Denable_dnn=enabled \ -Dc_args=-fprofile-update=atomic -Dcpp_args=-fprofile-update=atomic ninja -C core/build-coverage LD_LIBRARY_PATH=/path/to/onnxruntime/lib \ meson test -C core/build-coverage --num-processes 1 cd core && gcovr --root .. --filter 'src/.*' \ --exclude '.*/test/.*' --exclude '.*/tests/.*' \ --json-summary build-coverage/coverage.json build-coverage bash scripts/ci/coverage-check.sh core/build-coverage/coverage.json 37 85 # Pre-audit: tiny_extractor_template.h reports "min 10%" # Post-audit: same file reports "min 75%" — gate enforces the new floor Closes the ADR-0114 implicit follow-up "re-audit slack overrides on a cadence so they don't rot". Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(ci,operator): install kubebuilder envtest binaries for operator suite The cmd/vmafx-operator/internal/controller envtest suite was hard-failing in BeforeSuite with a nil-pointer deref from controlplane.(*APIServer).Stop because the kubebuilder envtest control-plane binaries (etcd + kube-apiserver + kubectl) were not on PATH. PRs #330, #341, and #362 all called this out as a pre-existing failure they could not address inline. Three-pronged fix: 1. Makefile gains a `setup-envtest` target that installs `sigs.k8s.io/controller-runtime/tools/setup-envtest@latest` and downloads the v1.31 control-plane bundle. A companion `setup-envtest-env` target prints the eval-friendly export line so developers can do `eval $(make -s setup-envtest-env)`. 2. .github/workflows/go-ci.yml installs setup-envtest and exports `KUBEBUILDER_ASSETS` via `$GITHUB_ENV` before `go test ./...`, so the operator suite runs for real in CI instead of skipping. 3. cmd/vmafx-operator/internal/controller/suite_test.go gains a top-of-`TestControllers` `t.Skip()` guard when `KUBEBUILDER_ASSETS` is unset, plus a nil-`testEnv` bailout in `AfterSuite` so the suite never panics on a fresh checkout where the assets are missing (defense in depth). Local verification: - `unset KUBEBUILDER_ASSETS && go test ./cmd/vmafx-operator/...` -> SKIP with an actionable message pointing at `make setup-envtest`. - `eval $(make -s setup-envtest-env) && go test -v ./cmd/vmafx-operator/...` -> 3/3 specs pass in ~5 s. AGENTS.md gains a new invariant #6 documenting the skip-safe envtest pattern; rebase-notes.md, state.md, and a changelog fragment under changelog.d/fixed/ are updated per ADR-0108 + CLAUDE.md r10/r13. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * test(sycl): SYCL kernel parity coverage round 3 (5 new tests, ADR-0946) Adds five new CPU-vs-SYCL parity tests under core/test/, extending the SYCL parity coverage beyond rounds 1 (PR #351: integer psnr + vif) and 2 (PR #376: integer adm / ciede / ssim / ms_ssim / motion_v2). New parity tests at ADR-0214 places=4 (1e-4) tolerance: | Kernel | New test | Headline score | |---------------------|---------------------------------------|-------------------------------------------| | float_psnr_sycl | test_sycl_float_psnr_parity.c | float_psnr | | float_adm_sycl | test_sycl_float_adm_parity.c | VMAF_feature_adm2_score | | float_vif_sycl | test_sycl_float_vif_parity.c | VMAF_feature_vif_scale0_score | | float_motion_sycl | test_sycl_float_motion_parity.c | VMAF_feature_motion2_score (idx 1) | | psnr_hvs_sycl | test_sycl_psnr_hvs_parity.c | psnr_hvs | Each mirrors the round-1 / round-2 pattern: 256x144 synthetic YUV420P fixture, CPU + SYCL feature extractor, parity assertion within ADR-0214 places=4, skip-on-no-device via "[skip: no SYCL device]" printf. Coverage trajectory: 50% (rounds 1+2) -> 78% (this PR). Round-4 backlog: float_moment, speed_chroma, speed_temporal, ssimulacra2 (need scaffold extensions for per-extractor config dicts and ref_pic_90 fill before the same gate can be added). Container compile-check evidence: docker exec vmaf-dev-mcp bash -lc ' source /opt/intel/oneapi/setvars.sh --force >/dev/null 2>&1 && \ cd /tmp/wt-sycl-r3 && \ CC=icx CXX=icpx meson setup build-sycl-r3 core \ -Denable_sycl=true -Denable_avx512=false -Db_lto=false && \ ninja -C build-sycl-r3 \ test/test_sycl_float_psnr_parity \ test/test_sycl_float_adm_parity \ test/test_sycl_float_vif_parity \ test/test_sycl_float_motion_parity \ test/test_sycl_psnr_hvs_parity' All five executables compile + link clean with -Wall -Wextra under icx/icpx. Registration sub-tests pass on this host; parity sub-tests hit a pre-existing level_zero device-passthrough issue (same as PR #376, not introduced by this PR). Refs ADR-0946, ADR-0214, ADR-0868, ADR-0884. * test(sycl): SYCL kernel parity coverage round 4 (4 new tests, ADR-0957) (#465) Closes the SYCL kernel-coverage backlog enumerated in ADR-0946 by adding four new CPU vs. SYCL parity tests under core/test/. | Kernel | New test | Tolerance | |---------------------|---------------------------------------|------------------------| | float_moment_sycl | test_sycl_float_moment_parity.c | 1e-4 (ADR-0214 default)| | speed_chroma_sycl | test_sycl_speed_chroma_parity.c | 1e-4 (ADR-0214 default)| | speed_temporal_sycl | test_sycl_speed_temporal_parity.c | 1e-4 (ADR-0214 default)| | ssimulacra2_sycl | test_sycl_ssimulacra2_parity.c | 5e-3 (FEATURE_TOLERANCE)| Each test mirrors the round-3 scaffold: 256x144 synthetic YUV420P fixture, public vmaf_use_feature API with NULL options dict (defaults match between CPU and SYCL for all four kernels), parity assertion via fabs(cpu - sycl) <= TOL, skip-on-no-device via "[skip: no SYCL device]" printf. The SSIMULACRA2 fixture fills all three planes (the pipeline consumes YUV -> linear-RGB -> XYB and chroma matters for the headline score). The speed_temporal fixture submits two frames (TEMPORAL flag means frame 0 emits 0.0) and asserts at index 1. Discovery during round-4 implementation: speed_chroma_sycl.cpp (752 LOC) and speed_temporal_sycl.cpp (705 LOC) source files exist on disk but are NOT wired into sycl_feature_sources in core/src/meson.build and their extractor symbols are NOT declared/registered in core/src/feature/feature_extractor.c. Both files appear complete (no TODO/FIXME/-ENOSYS/stub markers) but ship as dormant scaffold. Wiring them in is out of scope for a kernel-coverage PR (it changes the production extractor surface). The two SpEED parity tests are added in dormant form with a "[skip: <name> not built into libvmaf]" guard that auto-activates as a real parity gate the day a follow-up PR wires the TUs into the build + registry. Container compile-check evidence (CC=icx CXX=icpx, enable_sycl=true, Intel Arc A380 visible): - All four test executables link against libvmaf.a successfully. - speed_chroma/temporal: pass (skip — dormant scaffold). - float_moment / ssimulacra2: register-existence sub-test passes; parity sub-test hits the same pre-existing level_zero device-passthrough SIGSEGV that PR #446's round-3 tests hit on this dev container (not introduced by this PR). On a host with proper Intel-GPU passthrough or with no SYCL device, the tests pass / skip cleanly. Coverage trajectory: - Round 0 (pre-rounds): 2/18 (11%) - Round 1 (#351): +2 = 4/18 (22%) - Round 2 (#376): +5 = 9/18 (50%) - Round 3 (#446): +5 = 14/18 (78%) - Round 4 (this PR): +4 = 18/18 (100% of round-3 backlog; 16 active + 2 dormant SpEED scaffolds) Refs ADR-0957, ADR-0214, ADR-0867, ADR-0884, ADR-0946. Co-authored-by: Lusoris <lusoris@pm.me> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> * chore(ci): add changelog fragment for infra/CI bundle batch-1 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Lusoris <lusoris@pm.me> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>

lusoris marked this pull request as ready for review May 31, 2026 12:59

lusoris closed this May 31, 2026

lusoris deleted the chore/coverage-overrides-audit branch May 31, 2026 13:15

lusoris restored the chore/coverage-overrides-audit branch May 31, 2026 18:39

lusoris reopened this May 31, 2026

lusoris marked this pull request as draft May 31, 2026 18:48

lusoris mentioned this pull request Jun 1, 2026

chore(ci): infra/CI bundle batch-1 (#370 coverage override + #396 envtest + #446 SYCL parity round 3) #528

Merged

7 of 9 tasks

lusoris closed this Jun 1, 2026

lusoris deleted the chore/coverage-overrides-audit branch June 4, 2026 08:08

Navigation Menu

Repository navigation

chore(ci): audit per-file coverage overrides — tighten tiny_extractor_template.h 10 → 75 - #370

chore(ci): audit per-file coverage overrides — tighten tiny_extractor_template.h 10 → 75#370
lusoris wants to merge 1 commit into
masterfrom
chore/coverage-overrides-audit

lusoris commented May 30, 2026

Uh oh!

lusoris commented May 31, 2026

Uh oh!

lusoris commented Jun 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

lusoris commented May 30, 2026

Summary

Per-override action table

New files surveyed (none flagged)

Codified audit rule (lands in ADR-0881)

Policy guard

Reproducer

Type

Bug-status hygiene (ADR-0165)

Netflix golden-data gate (ADR-0024)

Deep-dive deliverables (ADR-0108)

Uh oh!

lusoris commented May 31, 2026

Uh oh!

lusoris commented Jun 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant