ci: enforce per-package coverage threshold via vitest [closes #218]#276
Open
EmersonBraun wants to merge 1 commit intomainfrom
Open
ci: enforce per-package coverage threshold via vitest [closes #218]#276EmersonBraun wants to merge 1 commit intomainfrom
EmersonBraun wants to merge 1 commit intomainfrom
Conversation
Adds @vitest/coverage-v8 and a shared test config helper enforcing a 'lines' coverage threshold per package on every PR. Architecture: - vitest.shared.ts at root: createTestConfig(opts) helper with v8 provider, per-package linesThreshold, optional environment + setupFiles - Each package's vitest.config.ts now imports + extends the helper - New 'test:coverage' script per package runs vitest with --coverage - New top-level 'pnpm test:coverage' via turbo - New .github/workflows/coverage.yml runs on every PR + push to main Per-package thresholds (current minus 5% buffer; tighten as coverage grows): Package Actual Threshold Headroom ------- ------ --------- -------- rag 100% 95 +5 skills 100% 95 +5 eval 100% 95 +5 templates 100% 95 +5 runtime 94.33% 90 +4 react 86.60% 80 +6 memory 84.02% 80 +4 core 79.25% 75 +4 (sacred — target 90%) tools 76.00% 70 +6 adapters 63.83% 60 +4 observability 57.69% 55 +3 cli 36.70% 30 +7 sandbox 35.38% 30 +5 Why lines-only as the gate metric: simplest signal, most stable across test styles. Branches/functions/statements still reported but not blocking. Aspirational targets for next 2 sprints: - Every package ≥ 80% lines - @agentskit/core ≥ 90% lines (Manifesto principle 1) Closes #218
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds vitest coverage gate to every package via shared config helper. Closes #218. Pairs with #275 (size-limit) to formalize quality bars.
Architecture
vitest.shared.tsat root —createTestConfig({ linesThreshold, environment?, setupFiles? })helper with v8 providervitest.config.tsnow imports + extends the helper (one-line config per package)test:coveragescript per package runsvitest run --coveragepnpm test:coveragevia turbo.github/workflows/coverage.ymlruns on every PR + push to main; blocks merge on regression below thresholdPer-package thresholds (current minus 5% buffer)
@agentskit/rag@agentskit/skills@agentskit/eval@agentskit/templates@agentskit/runtime@agentskit/react@agentskit/memory@agentskit/core@agentskit/tools@agentskit/adapters@agentskit/observability@agentskit/cli@agentskit/sandboxWhy lines-only as the gate metric
Simplest signal, most stable across test styles. Branches/functions/statements still reported (text + html + lcov + json-summary) but not blocking. Avoids the failure mode where 100% line coverage with one branch missed produces a noisy red.
Aspirational targets (next 2 sprints)
@agentskit/core≥ 90% lines (Manifesto principle 1 — sacred)These are not enforced; they are the goalposts. The thresholds in this PR are the floor that prevents regression.
Trade-offs
index.tsfiles excluded from coverage (re-exports only). Source files stay in scope.@agentskit/inktests were broken onmainbefore #265 (ink@7 / ink-testing-library@4 incompat). This PR's CI will fail on the ink package until #265 merges. Other 13 packages pass coverage threshold locally.Recommended order: merge #265 → rebase this PR → CI green.
Test plan
pnpm --filter <pkg> test:coverage)pnpm test:coverageruns all in parallel via turbohappy-domenv +setupFilesvia helper optsFollow-ups
Closes #218
Refs #211