Repository navigation
test(coverage): push critical-file coverage above ADR-0922 ratchet floors - #508
Merged
Merged
Conversation
…oors PR #469 ratcheted the Coverage Gate floors: - OVERALL_MIN: 37% -> 70% - CRITICAL_MIN: 85% -> 90% - PER_FILE_MIN[dnn_api.c]: 78% -> 83% - PER_FILE_MIN[ort_backend.c]: 78% -> 83% - PER_FILE_MIN[tiny_extractor_template.h]: 10% -> 15% + new per-PR delta gate (scripts/ci/coverage-delta-check.sh) The last successful master CI run (21597ff, 2026-05-31 10:53) reported read_json_model.c at 88.4% and model_loader.c at 86.9% — both now below the new 90% critical floor. This PR adds 31 unit tests targeting the under-floor paths. After this PR (unit-test-only measurement, container build matches CI flags: -Db_coverage=true -Denable_float=true -Denable_avx512=true -Denable_dnn=enabled -Dc_args=-fprofile-update=atomic): core/src/read_json_model.c 88.00% -> 92.00% (+4.00pp) PASS core/src/dnn/model_loader.c 87.20% -> 90.00% (+2.80pp) PASS core/src/dnn/dnn_api.c 83.40% (unchanged) PASS core/src/dnn/dnn_attach_api.c 92.00% (unchanged) PASS core/src/dnn/ort_backend.c 78.90% (unchanged) FAIL (*) core/src/dnn/onnx_scan.c 93.40% (unchanged) PASS core/src/dnn/op_allowlist.c 100.00% (unchanged) PASS core/src/dnn/tensor_io.c 99.30% (unchanged) PASS core/src/opt.c 100.00% (unchanged) PASS (*) ort_backend.c's remaining uncovered lines are ORT-API error-status branches and non-CPU execution-provider success branches. On CPU-only ORT builds (current CI runner) the EP try_append_{cuda,openvino,coreml, rocm} calls all return non-zero and the conditional `sess->ep_name = ...` assignments are unreachable; the GetTensorElementType-failure / calloc- failure branches require ORT-API mock injection. The structural ceiling on this configuration is ~79%. Pushing above 83% needs either: (a) ORT-API mocking via the ort_backend_internal.h test hook, or (b) a working CUDA / OpenVINO EP on the CI runner. New tests by file: core/test/test_model.c (+17 tests): test_json_model_slopes_not_array test_json_model_intercepts_not_array test_json_model_feature_names_not_array test_json_model_feature_opts_dicts_not_array test_json_model_model_payload_not_string test_json_model_chroma_correction_not_number test_json_model_score_clip_min_not_number test_json_model_score_clip_max_not_number test_json_model_score_transform_poly_null_disables test_json_model_score_transform_knots_null_disables test_json_model_score_transform_bool_str_not_string test_json_model_unrecognised_model_dict_key test_json_model_intercepts_mid_not_number test_json_model_score_transform_poly_number_sets_value test_json_model_score_transform_poly_string_rejects test_json_model_score_transform_knots_array_walks core/test/dnn/test_model_loader.c (+13 tests): test_sidecar_encoder_vocab_malformed_no_leak test_sidecar_empty_arrays_are_valid test_sidecar_encoder_vocab_over_max_returns_erange test_sidecar_array_trailing_junk_wipes test_sidecar_array_non_string_element_wipes test_sidecar_opset_overflow_returns_default test_sidecar_feature_mean_trailing_junk test_sidecar_quant_mode_static test_sidecar_quant_mode_qat test_sidecar_kind_filter test_codec_block_fill_aliases_hevc_av1_vp9_vvc test_codec_block_fill_preset_slower test_codec_block_fill_null_vocab_entry_is_skipped core/test/dnn/test_ort_internals.c (+1 test): test_ort_run_multi_output_smoke Verification: meson test -C core/build-coverage --num-processes 1 -> 85/85 PASS (including all 31 new tests) Each new test exercises at least one previously-uncovered branch identified from the CI master-tip gcovr txt summary (run 26710610282, sha 21597ff). No production code changed; no golden assertions or places= tolerances touched. This PR does NOT close the OVERALL 70% floor (current unit-test-only measurement at 47.6%). CI's overall measurement is higher because the Coverage Gate job also runs the full Python test suite under the instrumented libvmaf.so / vmaf CLI. Pushing the unit-test-only overall above 70% would require porting ~300 Python tests into native C tests — out of scope for a single PR. The last successful master CI showed overall at 45.6% — the new 70% floor will trip on master itself until the per-file ratchet's measurement assumptions are reconciled. Closes part of the PR #469 ratchet gap; ort_backend.c + overall remain follow-ups. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PR #469 ratcheted Coverage Gate floors (ADR-0922):
OVERALL_MIN37→70 %,CRITICAL_MIN85→90 %, plus per-file floors ondnn_api.c/ort_backend.c(78→83 %) andtiny_extractor_template.h(10→15 %). Also added a per-PR delta gate (no touched file may drop > 0.5pp). The last successful master CI run measured two critical files below the new floor:read_json_model.cat 88.4 % vs new 90 %, andmodel_loader.cat 86.9 % vs new 90 %.This PR adds 31 targeted unit tests that push both files above their new floors. No production code touched; no golden assertions or
places=tolerances modified.Type
test— test-onlyCoverage delta (unit-test-only, container build matching CI flags)
core/src/read_json_model.ccore/src/dnn/model_loader.ccore/src/dnn/dnn_api.ccore/src/dnn/dnn_attach_api.ccore/src/dnn/onnx_scan.ccore/src/dnn/op_allowlist.ccore/src/dnn/ort_backend.ccore/src/dnn/tensor_io.ccore/src/dnn/tiny_extractor_template.hcore/src/opt.c(*)
ort_backend.cremaining uncovered lines split into two structural categories on CPU-only ORT builds:sess->ep_name = "..."assignments inside the EP-selector switch never fire because eachtry_append_<ep>returns non-zero on a CPU-only build.GetTensorElementType,CreateTensorWithDataAsOrtValue,Run) — require either ORT-API mock injection or actual ORT misbehaviour, neither reachable from a happy-path unit test.The structural ceiling on the current CPU-only ORT CI runner is approximately 79 %. Pushing above the 83 % floor needs (a) ORT-API mock injection via
ort_backend_internal.h, or (b) a working CUDA / OpenVINO EP on the runner — both follow-up scope.Tests added by file
core/test/test_model.c(+17 read_json_model.c branch tests):-EINVALbranches acrossparse_model_dict_array_key/parse_model_dict_chroma_correction/parse_model_dict_score_clip(slopes/intercepts/feature_names/feature_opts_dicts/model not array or string, score_clip min/max not number)parse_score_transform_polybranches (JSON_NULL disables, NUMBER sets value, STRING rejects)parse_score_transform_knots_keyNULL-disable and array-walk pathsparse_score_transform_bool_strnon-string rejectparse_interceptsmid-loop type-mismatch (after first valid entry)parse_model_dict_entryunrecognised-key skip pathcore/test/dnn/test_model_loader.c(+13 model_loader.c branch tests):extract_string_arrayADR-0976 leak-free contract forencoder_vocab(third call site afteroutput_names+feature_order)extract_string_arrayempty-array happy path (line 211 — was unreachable through any existing fixture)extract_string_array-ERANGEover-cap (encoder_vocab with 33 entries vs 32 cap)extract_string_arraytrailing-junk-EINVAL(non-comma non-]after a valid entry)extract_string_arraynon-string element-EINVAL(output_names: [42])extract_int-ERANGE(onnx_opset: 99999999999999999999999→ default 0)extract_float_arraytrailing-junk path viafeature_mean: [1.5 Z]quant_modeliteral-string branches:"static","qat"(existing tests covered"dynamic"+ unknown-fallback only)kind: "filter"literal-string branch (existing tests covered"fr"+"nr"only)resolve_codec_aliasbranches:hevc/h265→libx265,av1→libsvtav1,vp9→libvpx-vp9,vvc/h266→libvvenc,avc→libx264(existing tests coveredh264→libx264only)slowerpreset ordinal (the only preset name not in the existing preset-table grid)vmaf_dnn_codec_block_fillNULL-vocab-entry skip branchcore/test/dnn/test_ort_internals.c(+1 ort_backend.c branch test):vmaf_ort_runmulti-output happy-path drives the per-outputcopy_output_tensor()+ReleaseValue()loops withn_outputs > 1using the shippedmodel/tiny/smoke_multi_output_v0.onnxfixtureReproducer
Bug-status hygiene
no state delta: REASON— pure test-only PR; no bug closed, opened, or ruled out.Netflix golden-data gate
assertAlmostEqual(...)score in the Netflix golden Python tests.Known follow-ups
ort_backend.c78.9 % vs 83 % floor — structural ceiling on CPU-only ORT builds. Requires either ORT-API mock injection (via existingort_backend_internal.hhook surface) or a working non-CPU EP on the runner. Out of scope for a unit-test-only PR.21597ff8) comes from instrumenting both the C unit suite and the full Python test suite under the instrumentedlibvmaf.so/vmafCLI. Closing the OVERALL gap by porting ~300 Python tests into native C tests is multi-PR scope. The 70 % floor will likely trip on master itself on the next successful CI run until either the floor is reconciled with the measured value or the Python-suite Coverage Gate path lands additional tests.Deep-dive deliverables (ADR-0108)
AGENTS.mdinvariant note — no rebase-sensitive invariants (test-only additions to existing fork-added test files; no upstream-mirror surface touched).changelog.d/added/coverage-push-pr469-floors.md.🤖 Generated with Claude Code