Repository navigation
Tune skill descriptions and eval cases to a reliable trigger rate, and document how #2202
Copy link
Copy link
Closed
Labels
agent guidanceRules and skills for agentsRules and skills for agentsdocumentationImprovements or additions to documentationImprovements or additions to documentationv2Issues and PRs for v2Issues and PRs for v2
Milestone
Description
npm run skills:eval was passing 38 of 46 cases, with several model-invoked skills sitting at 0–33% trigger rates. Investigating showed the low scores were mostly bad eval cases rather than bad descriptions, and that the reasons why are not written down anywhere — so the next person to add a skill will rediscover them.
Two root causes, both invisible from the eval output alone:
- The harness runs each case with
--max-turns 1. The skill has to be the model's first tool call. A prompt phrased as "how does X work here" or naming a concrete file sends the model toRead/Grepfirst — a better answer to the question, scored as a miss. Every 0% case in the suite was one of these. AGENTS.mdis loaded in full on every turn. A prompt it already answers gets answered without any skill, so the case can never pass. Three committed cases were in this category, including one that had been at 0% across every run.
What this issue covers
- Replace the cases that measure the wrong thing, and expand every model-invoked skill from 3 to 5 positive cases (negatives kept), so a single flake cannot cross the threshold.
- Reshape the
local-devdescription to the action-first "Use when …; when …" form that measurement keeps rewarding. - Add a
docs/page describing how to write a skill description that fires and eval cases that measure it, so this is a reference instead of a re-derivation.
Acceptance
RUNS=5 npm run skills:eval— the whole suite, not just the edited skills — at ≥80% on every case, with all negative cases clean.- Every model-invoked skill carries 5 positive cases plus its negatives, and no committed case is answerable from
AGENTS.mdalone. - A new
docs/page captures everything this investigation established, linked from the README documentation table and fromAGENTS.md's skill-maintenance rules, and covering at minimum:- How the eval actually measures — fresh session,
--max-turns 1, negatives scored against every model-invoked skill in the repo, positives restricted to their own skill. - The first-move rule: the skill must be the model's first tool call, so "how does X work here" and any prompt naming a concrete file lose the turn to
Read/Grep. With the measured before/after prompt pairs. - The case shapes that reliably fire, and the ones that reliably do not.
- Why a case
AGENTS.mdalready answers can never pass, with the retired cases as worked examples. - When a neighbouring skill fires and is arguably right, the case is wrong — not the description.
- Description shape: action-first, then
Use when …; when …; when …. - The tuning loop — probe first (and probe a marginal prompt more than once), then measure; how much noise to expect at
RUNS=3vsRUNS=5; and why this can never be a CI gate. - A checklist for adding or editing a skill.
- How the eval actually measures — fresh session,
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
agent guidanceRules and skills for agentsRules and skills for agentsdocumentationImprovements or additions to documentationImprovements or additions to documentationv2Issues and PRs for v2Issues and PRs for v2
Activity
cliffhall commentedon Sep 1, 2026
Triage: Priority Medium (total 6)
bug, no engagement yet, not assigned, not a sub-issue. The milestone bonus is deliberately not claimed — this issue was filed through the create flow, so its milestone is the approval act rather than corroborating signalBoard: #28, Status
In Progress.docs: tune skill eval cases and document how to write them