Skip to content

ci: give the Codecov gate a target it can actually fail - #148

Merged
rmems merged 1 commit into
mainfrom
ci/real-codecov-thresholds
Sep 8, 2026
Merged

rmems merged 1 commit into
mainfrom
ci/real-codecov-thresholds

Conversation

@rmems

@rmems rmems commented Sep 7, 2026 •

Copy link
Copy Markdown
Owner

User description

The defect

.github/codecov.yml configured both statuses like this:

project:
  default:
    target: 0
    threshold: 0.0
patch:
  default:
    target: 0
    threshold: 0.0

A target of zero means "require 0% coverage." Every possible report satisfies it. Coverage could fall from 86% to nothing and both checks would still report green. The statuses appeared on every PR and measured none of them.

The fix

status before after meaning
project target: 0 target: auto, threshold: 1% may not drop >1 point vs. the base commit
patch target: 0 target: 80%, threshold: 0% new/changed lines must be ≥80% covered

Both numbers are grounded in what this workspace already delivers, so neither asks for more than the status quo:

80% sits below both. project uses auto so the gate keeps working as coverage rises; patch uses a fixed floor so a contributor knows exactly what new code must clear rather than chasing a number that drifts.

require_ci_to_pass stays false — deliberately

Now with the reason recorded in the file. Codecov cannot distinguish the native Rust gates from the several third-party review bots that also report checks on this repo, so true lets an unrelated bot hold the coverage status pending indefinitely. A genuinely broken build is already caught by test (ubuntu-latest) going red, which blocks the PR regardless.

Also

Drops the **/.venv/** ignore entry, left over from the removed Python layer.

Verification

POST https://codecov.io/validate returns Valid! and echoes the parsed targets:

"project": { "default": { "target": "auto", "threshold": 1.0 } },
"patch":   { "default": { "target": 80.0,   "threshold": 0.0 } }

Refs #1

🤖 Generated with Claude Code

https://claude.ai/code/session_016yFLnKN9zQA7uzgpYxTrjX


CodeAnt-AI Description

Enforce meaningful Codecov coverage checks for projects and changed lines

What Changed

  • Project coverage must stay within 1 percentage point of the base commit.
  • New and modified coverable lines must reach at least 80% coverage.
  • Codecov checks remain independent of unrelated third-party status checks, while build failures continue to block through the Rust test workflow.
  • Removes the obsolete virtual-environment exclusion and defines the coverage range shown in the Codecov UI.

Impact

✅ Coverage regressions can fail pull requests
✅ New code must meet an 80% coverage floor
✅ Unrelated review bots cannot leave coverage checks pending

💡 Usage Guide

Checking Your Pull Request

Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.

Talking to CodeAnt AI

Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:

@codeant-ai ask: Your question here

This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.

Example

@codeant-ai ask: Can you suggest a safer alternative to storing this secret?

Preserve Org Learnings with CodeAnt

You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:

@codeant-ai: Your feedback here

This helps CodeAnt AI learn and adapt to your team's coding style and standards.

Example

@codeant-ai: Do not flag unused imports.

Retrigger review

Ask CodeAnt AI to review the PR again, by typing:

@codeant-ai: review

Check Your Repository Health

To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.


Summary by cubic

Fixes the Codecov project and patch gates so coverage regressions can actually fail the PR; with target: 0 both statuses required 0% coverage and every report passed, from 86% down to nothing.

  • Project gate now uses target: auto with a 1% threshold, tracking the base commit so the gate keeps working as coverage rises.
  • Patch gate now requires 80% coverage on new lines as a fixed floor; the 86% project and 100% patch baselines mean this asks no more than the codebase already delivers.
  • Keeps require_ci_to_pass false because Codecov can't distinguish native CI gates from third-party review-bot checks; the test (ubuntu-latest) job already blocks broken builds.
  • Removes the stale **/.venv/** ignore from the removed Python layer.

Refs #1

Written for commit 4e0c9c7. Summary will update on new commits.

Review in cubic

Both Codecov statuses were configured with `target: 0` and
`threshold: 0.0`. A target of zero means "require 0% coverage", which
every possible report satisfies. Coverage could fall from 86% to nothing
and the project and patch checks would both still report green. The
statuses were present on every PR and measured none of them.

Replaced with thresholds grounded in what this workspace already does,
so neither asks for more than it currently delivers:

  project: target auto, threshold 1%
    Tracks the base commit instead of a fixed number, so the gate keeps
    working as coverage rises. 1% absorbs line-count noise, not a real
    regression.

  patch: target 80%, threshold 0%
    A fixed floor rather than `auto`, so a contributor knows exactly what
    new code must clear and it does not tighten unpredictably as project
    coverage drifts.

Baselines, on main @ 00d6989: project coverage 86%; PR #144 patch
coverage 100% ("All modified and coverable lines are covered by tests").
80% sits below both.

`require_ci_to_pass` stays false, now with the reason recorded in the
file: Codecov cannot tell the native Rust gates apart from the several
third-party review bots that also report checks here, so `true` lets an
unrelated bot hold the coverage status pending. A genuinely broken build
is already caught by `test (ubuntu-latest)` going red.

Also drops the `**/.venv/**` ignore, left over from the Python layer.

Verified: `POST https://codecov.io/validate` returns "Valid!" and echoes
the parsed targets.

Refs #1

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016yFLnKN9zQA7uzgpYxTrjX
@codeant-ai

codeant-ai Bot commented Sep 7, 2026 •

Copy link
Copy Markdown

🤖 CodeAnt AI — Review Status

Status Commit Started (UTC) Finished (UTC)
✅ Reviewed your PR 4e0c9c7 Sep 07, 2026 · 07:15 07:17

@codeant-ai

codeant-ai Bot commented Sep 7, 2026

Copy link
Copy Markdown

Thanks for using CodeAnt! 🎉

We're free for open-source projects. if you're enjoying it, help us grow by sharing.

Share on X ·
Reddit ·
LinkedIn

@coderabbitai

coderabbitai Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

Next included review available in 58 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Team

Run ID: 8f1c2145-e901-494d-b737-b16052f3c23d

📥 Commits

Reviewing files that changed from the base of the PR and between 00d6989 and 4e0c9c7.

📒 Files selected for processing (1)
  • .github/codecov.yml

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@codeant-ai codeant-ai Bot added the size:M This PR changes 30-99 lines, ignoring generated files label Sep 7, 2026

@codescene-access codescene-access Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No application code in the PR — skipped Code Health checks.

See analysis details in CodeScene

Quality Gate Profile: Pay Down Tech Debt
Install CodeScene MCP: safeguard and uplift AI-generated code. Catch issues early with our IDE extension and CLI tool.

@amazon-q-developer amazon-q-developer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Codecov configuration changes are well-structured and properly validated. The move from target: 0 (which effectively disabled coverage gates) to meaningful thresholds (auto for project with 1% tolerance, and 80% for patch coverage) establishes actual quality gates grounded in current metrics. The extensive inline documentation clearly explains the rationale for each setting, and the YAML syntax is correct.


You can now have the agent implement changes and create commits directly on your pull request's source branch. Simply comment with /q followed by your request in natural language to ask the agent to make changes.

@codacy-production

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@codecov

codecov Bot commented Sep 7, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@rmems rmems assigned rmems and Claude Sep 8, 2026
@rmems
rmems merged commit 9d6aaf9 into main Sep 8, 2026
16 checks passed
@rmems
rmems deleted the ci/real-codecov-thresholds branch September 8, 2026 03:40
@linear-code

linear-code Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

LIM-1078

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

github-actions size:M This PR changes 30-99 lines, ignoring generated files

Projects

Development

Successfully merging this pull request may close these issues.

2 participants