Skip to content

chore(ci): ratchet Coverage Gate floors + add per-PR delta gate (ADR-0922) - #421

Closed
lusoris wants to merge 0 commit into
masterfrom
chore/coverage-ratchet-aggressive
Closed

lusoris wants to merge 0 commit into
masterfrom
chore/coverage-ratchet-aggressive

Conversation

@lusoris

@lusoris lusoris commented May 31, 2026

Copy link
Copy Markdown
Contributor

Summary

Raise the Coverage Gate's absolute floors aggressively and introduce a new
per-PR coverage-delta gate so coverage can no longer decay one PR at a
time between absolute-floor trips. See
ADR-0922
for the full rationale, the alternatives table, and the 30-day grace
window for in-flight PRs.

Floor changes (one-way ratchet)

Threshold Old New File
OVERALL_MIN 37 % 60 % scripts/ci/coverage-check.sh
CRITICAL_MIN 85 % 90 % scripts/ci/coverage-check.sh
PER_FILE_MIN["core/src/dnn/ort_backend.c"] 78 % 83 % scripts/ci/coverage-check.sh
PER_FILE_MIN["core/src/dnn/dnn_api.c"] 78 % 83 % scripts/ci/coverage-check.sh
PER_FILE_MIN["core/src/dnn/tiny_extractor_template.h"] 10 % 15 % scripts/ci/coverage-check.sh
Delta-gate overall drop tolerance n/a 0.5pp scripts/ci/coverage-delta-check.sh
Delta-gate per-touched-file drop tolerance n/a 0.5pp scripts/ci/coverage-delta-check.sh

Lowering any floor or loosening the delta tolerance requires a new ADR
that explicitly supersedes ADR-0922 and is cited inline at the changed
threshold.

New gate: scripts/ci/coverage-delta-check.sh

Compares head vs. merge-base gcovr summaries on pull_request events and
fails if:

  • overall coverage drops by more than --max-overall-drop (default
    0.5pp), OR
  • any file present in both reports AND touched by the PR's diff drops by
    more than --max-file-drop (default 0.5pp).

New files have no base row to compare to and are covered by the absolute
floors only. Files not touched by the PR are not scored. Wired into the
existing coverage job in tests-and-quality-gates.yml; needs
actions/checkout with fetch-depth: 0 so git merge-base can resolve.

Grace window for in-flight PRs

PRs opened before 2026-05-31 are exempt from the new floors and the
delta gate for 30 days (through 2026-06-30). Reviewers apply the
exemption manually by checking the PR open-date; the exemption
self-disables on 2026-06-30 and the policy becomes uniform thereafter.

Type

  • ci — tooling / infra
  • docs — ADR + research digest + rebase note + changelog fragment

Checklist

  • Commits follow Conventional Commits.
  • pre-commit run --files <touched> is green locally (shfmt + shellcheck + ADR-0332 worktree guard + ADR-0386 collision + ADR-0105 header check + semgrep all pass).
  • No SIMD / GPU / C / Python code touched; no cross-backend or numerical impact.
  • If I added a new ADR, the ADR row lives in docs/adr/_index_fragments/0922-coverage-ratchet-aggressive.md and the slug is appended to docs/adr/_index_fragments/_order.txt.

Bug-status hygiene (ADR-0165)

  • no state delta: REASON — pure CI gate change, no bug-status impact.

Netflix golden-data gate (ADR-0024)

  • I did not modify any assertAlmostEqual(...) score in the Netflix golden Python tests.

Deep-dive deliverables (ADR-0108)

Reproducer

scripts/ci/coverage-delta-check.sh exercise (PASS + 2 FAIL cases) using
hand-crafted gcovr JSON fixtures:

# PASS case: overall +0.2pp, touched file foo.c +0.1pp.
cat > /tmp/base.json <<JSON
{
  "line_percent": 70.0,
  "files": [
    {"filename": "core/src/foo.c", "line_percent": 80.0, "line_total": 100, "line_covered": 80}
  ]
}
JSON
cat > /tmp/head.json <<JSON
{
  "line_percent": 70.2,
  "files": [
    {"filename": "core/src/foo.c", "line_percent": 80.1, "line_total": 100, "line_covered": 80},
    {"filename": "core/src/new.c", "line_percent": 60.0, "line_total": 30, "line_covered": 18}
  ]
}
JSON
printf 'core/src/foo.c\ncore/src/new.c\n' > /tmp/changed.txt
scripts/ci/coverage-delta-check.sh \
  --base-json /tmp/base.json --head-json /tmp/head.json --changed-files /tmp/changed.txt
# exit 0, "PASS: coverage-delta gate met"

# FAIL case: touched file dropped 2.0pp.
# exit 2, "FAIL: 1 touched file(s) dropped past the 0.5pp ratchet tolerance"
# (see commit message for the full 3-case smoke matrix)

scripts/ci/coverage-check.sh PASS-case smoke:

cat > /tmp/cov_pass.json <<JSON
{
  "line_percent": 65.0,
  "files": [
    {"filename": "core/src/dnn/ort_backend.c", "line_percent": 85.0, "line_total": 500, "line_covered": 425},
    {"filename": "core/src/dnn/dnn_api.c", "line_percent": 84.0, "line_total": 200, "line_covered": 168},
    {"filename": "core/src/dnn/tiny_extractor_template.h", "line_percent": 20.0, "line_total": 100, "line_covered": 20},
    {"filename": "core/src/dnn/onnx_scan.c", "line_percent": 95.0, "line_total": 100, "line_covered": 95},
    {"filename": "core/src/opt.c", "line_percent": 95.0, "line_total": 50, "line_covered": 47},
    {"filename": "core/src/read_json_model.c", "line_percent": 92.0, "line_total": 200, "line_covered": 184}
  ]
}
JSON
bash scripts/ci/coverage-check.sh /tmp/cov_pass.json 60 90
# exit 0, "PASS: coverage gate met (overall ≥60%, critical ≥90% where tested)"

Known follow-ups

  • Promote coverage-delta-check.sh smoke tests to a tracked test under
    scripts/ci/tests/ once the directory exists.
  • Plan a follow-up ADR to ratchet 60 -> 70 once two consecutive CI weeks
    show overall sitting above 65 %.
  • The docs/adr/README.md regeneration drift (fragments dir on master
    is incomplete) is out of scope for this PR; opted to manually append
    the ADR-0922 row instead of running the full concat script.

Generated with Claude Code

@lusoris
lusoris marked this pull request as ready for review May 31, 2026 02:51
@lusoris
lusoris enabled auto-merge (squash) May 31, 2026 02:51
@lusoris lusoris closed this May 31, 2026
auto-merge was automatically disabled May 31, 2026 02:51

Pull request was closed

@lusoris
lusoris force-pushed the chore/coverage-ratchet-aggressive branch from fdf2179 to aea2d2b Compare May 31, 2026 02:51
@lusoris
lusoris enabled auto-merge (squash) May 31, 2026 02:51
@lusoris

lusoris commented May 31, 2026

Copy link
Copy Markdown
Contributor Author

Re-opened as #469 (#421 force-pushed to master SHA by a broken auto-rebase script; this PR carries the original commit cleanly rebased onto current master). See #469

lusoris added a commit that referenced this pull request May 31, 2026
lusoris added a commit that referenced this pull request May 31, 2026
…0922) — recovery of #421 (#469)

* chore(ci): ratchet Coverage Gate floors + add per-PR delta gate (ADR-0922) — recovery of #421

* chore(ci): restore mcp-server test accidentally reverted by rebase

---------

Co-authored-by: Lusoris <lusoris@pm.me>
@lusoris
lusoris deleted the chore/coverage-ratchet-aggressive branch June 4, 2026 08:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant