Skip to content

feat(eval): add lightweight skill contract checks - #4

Open
Scared-Heart wants to merge 1 commit into
teambition:mainfrom
Scared-Heart:codex/add-eval-harness
Open

feat(eval): add lightweight skill contract checks#4
Scared-Heart wants to merge 1 commit into
teambition:mainfrom
Scared-Heart:codex/add-eval-harness

Conversation

@Scared-Heart

@Scared-Heart Scared-Heart commented May 25, 2026

Copy link
Copy Markdown

Summary

This PR adds a first lightweight eval layer for the Skill repository. The goal is to protect important Skill instruction contracts from accidental Markdown regressions, not to introduce full model-based evals yet.

Changes included:

  • add a zero-dependency Node.js runner for local contract checks
  • add initial cases for teambition and teambition-statistics
  • check key instruction contracts such as required env vars, pagination guidance, project inference, priority lookup, statistics tool order, and filterTql rules
  • document how to run and extend the checks
  • add a GitHub Actions workflow to run the checks on PRs and pushes to main

Scope

This is intentionally a small v0. It only reads local Markdown and JSON files. It does not:

  • call Teambition APIs
  • require real tokens or production data
  • run model-based evals
  • verify actual agent tool-call behavior

I think this still provides value as a baseline regression guard, because the current Skills are mostly instruction-driven and important behavior can regress through seemingly small Markdown edits.

Future direction

If this direction looks useful, the next layer could add mocked tool-call evals or optional integration-style evals against a test workspace. Those would be better suited for validating actual agent behavior, while this PR focuses on keeping the existing instruction contracts stable.

Verification

npm test

Result: 2/2 eval cases passed.

@Scared-Heart Scared-Heart changed the title feat(eval): add skill evaluation harness feat(eval): add lightweight skill contract checks May 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant