Skip to content

Tune skill descriptions and eval cases to a reliable trigger rate, and document how #2202

Closed

Description

@cliffhall

npm run skills:eval was passing 38 of 46 cases, with several model-invoked skills sitting at 0–33% trigger rates. Investigating showed the low scores were mostly bad eval cases rather than bad descriptions, and that the reasons why are not written down anywhere — so the next person to add a skill will rediscover them.

Two root causes, both invisible from the eval output alone:

  1. The harness runs each case with --max-turns 1. The skill has to be the model's first tool call. A prompt phrased as "how does X work here" or naming a concrete file sends the model to Read/Grep first — a better answer to the question, scored as a miss. Every 0% case in the suite was one of these.
  2. AGENTS.md is loaded in full on every turn. A prompt it already answers gets answered without any skill, so the case can never pass. Three committed cases were in this category, including one that had been at 0% across every run.

What this issue covers

  • Replace the cases that measure the wrong thing, and expand every model-invoked skill from 3 to 5 positive cases (negatives kept), so a single flake cannot cross the threshold.
  • Reshape the local-dev description to the action-first "Use when …; when …" form that measurement keeps rewarding.
  • Add a docs/ page describing how to write a skill description that fires and eval cases that measure it, so this is a reference instead of a re-derivation.

Acceptance

  1. RUNS=5 npm run skills:eval — the whole suite, not just the edited skills — at ≥80% on every case, with all negative cases clean.
  2. Every model-invoked skill carries 5 positive cases plus its negatives, and no committed case is answerable from AGENTS.md alone.
  3. A new docs/ page captures everything this investigation established, linked from the README documentation table and from AGENTS.md's skill-maintenance rules, and covering at minimum:
    • How the eval actually measures — fresh session, --max-turns 1, negatives scored against every model-invoked skill in the repo, positives restricted to their own skill.
    • The first-move rule: the skill must be the model's first tool call, so "how does X work here" and any prompt naming a concrete file lose the turn to Read/Grep. With the measured before/after prompt pairs.
    • The case shapes that reliably fire, and the ones that reliably do not.
    • Why a case AGENTS.md already answers can never pass, with the retired cases as worked examples.
    • When a neighbouring skill fires and is arguably right, the case is wrong — not the description.
    • Description shape: action-first, then Use when …; when …; when ….
    • The tuning loop — probe first (and probe a marginal prompt more than once), then measure; how much noise to expect at RUNS=3 vs RUNS=5; and why this can never be a CI gate.
    • A checklist for adding or editing a skill.

Activity

  1. added this to the v2.5.0 milestone on Sep 1, 2026
  2. added
    documentationImprovements or additions to documentation
    v2Issues and PRs for v2
    on Sep 1, 2026
  3. cliffhall commented on Sep 1, 2026

    @cliffhall
    MemberAuthor

    Triage: Priority Medium (total 6)

    • Severity 3 — a real capability was silently missing: two model-invoked skills measured 0% on situations they exist to cover, and a skill that cannot fire loses its behavior with no error anywhere
    • Urgency 3 — wanted this milestone; the work is done and the tuning loop is only useful written down before the next skill is added
    • Bonuses: none. Not a bug, no engagement yet, not assigned, not a sub-issue. The milestone bonus is deliberately not claimed — this issue was filed through the create flow, so its milestone is the approval act rather than corroborating signal

    Board: #28, Status In Progress.

  4. self-assigned this
    on Sep 1, 2026
  5. added a commit that references this issue on Sep 2, 2026
    6dd43e7
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

agent guidanceRules and skills for agentsdocumentationImprovements or additions to documentationv2Issues and PRs for v2

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions