triage skill: add a risk-severity rubric to prompts/triage.md (model systematically under-rates risk)

[don-petry]

Summary

The triage skill's held-out eval has regressed to 2/5 (score 0.4) as of the 2026-06-21 scheduled run (run 27901478735, tracked live in #814). All three failures share one root cause: the model under-rates risk by exactly one level, while the escalate boolean is always correct.

case	expected	got
triage-auth-token	escalate:true, HIGH	escalate:true, MEDIUM
triage-db-migration	escalate:true, HIGH	escalate:true, MEDIUM
triage-clean-logic-fix	escalate:false, MEDIUM	escalate:false, LOW

Root cause (verified)

prompts/triage.md rigorously defines when to escalate (criteria 1-7), which is why the escalate boolean is always right. But the risk field appears only in the output schema ("risk": "LOW|MEDIUM|HIGH") — there is no rubric defining what separates the three bands. The model is left to guess, and it guesses low.

scripts/evals/run-eval.sh (lines 161-162) requires both escalate and risk to match exactly, so a correct escalation with a one-level-low risk still fails the case.

This is not a label bug

The expected values are internally consistent across all 8 cases (5 holdout + 3 dev) under a single implicit rubric. Note triage-unresolved-thread (dev): it escalates but is MEDIUM, proving HIGH is reserved for dangerous content, not merely "escalated":

band	meaning (derived from the case data)	cases
HIGH	a criterion-1 high-risk area OR a criterion-3 security anti-pattern is present	auth-token, db-migration, sql-concat, workflow-pr-target
MEDIUM	clean non-trivial logic change, or escalation for process reasons (e.g. unresolved review thread)	clean-logic-fix, unresolved-thread
LOW	trivial change, no logic impact (docs, test-only)	docs-typo, test-only

This rubric predicts 8/8 expected labels. No holdout/dev case edits are warranted.

Proposed fix

Add a ## Risk rating section to prompts/triage.md that encodes the table above, anchored to the existing escalate criteria:

• HIGH — any criterion-1 high-risk area is touched, or a criterion-3 security anti-pattern is present.
• MEDIUM — escalation driven by process signals (unresolved threads, unaddressed advisory-bot findings, incomplete context), or a clean non-trivial logic/code change that does not escalate.
• LOW — trivial change with no logic impact (docs, comments, formatting, test-only additions).

Keep the change additive — one new section. Do not modify criteria 1-7 or the output schema; the escalate behavior is already correct and must stay untouched.

Acceptance criteria

• prompts/triage.md gains a ## Risk rating section defining HIGH/MEDIUM/LOW as above; no changes to the escalate criteria.
• Skill Eval Report re-run (via workflow_dispatch, skill=triage) scores 5/5 on the held-out set.
• Tracking issue Skill eval health — triage (holdout) #814 auto-closes on the recovered run.
• No changes to evals/triage/** case data.

References

• Live tracker: Skill eval health — triage (holdout) #814
• Skill prompt: prompts/triage.md
• Scorer: scripts/evals/run-eval.sh
• Held-out cases: evals/triage/holdout/cases.jsonl; dev cases: evals/triage/dev/cases.jsonl
• Workflow: .github/workflows/skill-eval-report.yml (daily 07:00 UTC, report-only / non-blocking)

[github-actions]

Thanks for the thorough write-up — the root-cause analysis, the 8/8 rubric derivation, and the clear before/after table make this extremely actionable! Before we add the ## Risk rating section, one quick question: are there any cases in the holdout or dev sets where the proposed HIGH/MEDIUM/LOW mapping feels ambiguous or borderline, or does the rubric hold cleanly across all 8 without edge cases we should explicitly encode?

[don-petry]

Dev-Lead Implementation Plan

Issue: #896 — triage skill: add a risk-severity rubric to prompts/triage.md (model systematically under-rates risk)

Scope

Add one additive ## Risk rating section to prompts/triage.md that encodes the HIGH/MEDIUM/LOW rubric anchored to the existing escalate criteria 1 & 3; no changes to criteria 1-7, the output schema, or any evals/triage/** case data.

Verification of the proposed rubric

Cross-checked the proposed bands against all 8 committed cases (5 holdout + 3 dev). The rubric predicts 8/8 expected labels, including triage-unresolved-thread (escalate:true but MEDIUM) and triage-clean-logic-fix (escalate:false but MEDIUM) — confirming HIGH is reserved for criterion-1 high-risk areas / criterion-3 anti-patterns, not merely "escalated". No case edits warranted.

Implementation Checklist

• prompts/triage.md — add ## Risk rating section (HIGH/MEDIUM/LOW), placed after the Decision criteria, before Output format; leave criteria 1-7 and the schema untouched
• tests/test_triage_prompt.bats — new deterministic structural guard asserting the Risk rating section exists and defines all three bands anchored to the criteria (regression guard against silent removal)
• .github/workflows/lint.yml — register the new bats file in the bats job list (required by AGENTS.md when adding test files)

Tests to Write

• tests/test_triage_prompt.bats: Risk rating heading present
• HIGH band references criterion-1 high-risk area / criterion-3 anti-pattern
• MEDIUM band references process signals + clean non-trivial change
• LOW band references trivial / docs / test-only changes
• escalate criteria section remains intact (no regression)

Note on behavioral validation

The held-out 5/5 acceptance criterion is validated by the live skill-eval-report workflow (workflow_dispatch, skill=triage), which requires the real Haiku-tier model and cannot run offline. The bats guard above is the deterministic, offline-runnable test; the live eval re-run is the behavioral confirmation.

Test Command

bats tests/test_triage_prompt.bats tests/test_skill_evals.bats tests/test_validate_cases.bats and bash .dev-lead/scripts/dev-lead-lint.sh

[don-petry]

Dev-Lead: Implementation Complete

Plan Execution

• Verified the proposed rubric against all 8 committed cases (5 holdout + 3 dev) → predicts 8/8 expected labels
• Added an additive ## Risk rating section to prompts/triage.md (HIGH/MEDIUM/LOW), anchored to escalate criteria 1 & 3; criteria 1-7 and the output schema left untouched
• Added tests/test_triage_prompt.bats — deterministic offline structural guard for the new section (TDD: written red, then green)
• Registered the new test in the bats job list in .github/workflows/lint.yml (per AGENTS.md)
• No changes to evals/triage/**

Test Results

$ bats tests/test_triage_prompt.bats
1..6
ok 1 triage prompt has a Risk rating section
ok 2 Risk rating section defines all three bands (HIGH, MEDIUM, LOW)
ok 3 HIGH band is anchored to criterion-1 high-risk areas or criterion-3 anti-patterns
ok 4 MEDIUM band covers process-driven escalation AND clean non-trivial changes
ok 5 LOW band is restricted to trivial / docs / test-only changes
ok 6 the escalate criteria (1-7) remain intact alongside the new section

$ bats test_triage_prompt + test_skill_evals + test_validate_cases + test_skill_eval_report
58 passing, 0 failing

$ bash .dev-lead/scripts/dev-lead-lint.sh
  [lint] shellcheck on 73 file(s)...
  [lint] validating 4 agent profile(s)...
  [lint] validating .github/CODEOWNERS...
  [lint] all checks passed

Files Changed

• prompts/triage.md: new ## Risk rating section defining HIGH/MEDIUM/LOW, each band anchored to the existing escalate criteria, plus a "highest band applies" tie-breaker that fixes the observed failure mode (a small criterion-1 diff — e.g. db-migration — being rated MEDIUM because the diff looked trivial).
• tests/test_triage_prompt.bats: new deterministic guard (section exists; all three bands present and correctly anchored; escalate criteria untouched). Section-scoped via awk so band checks can't match the schema's LOW|MEDIUM|HIGH token.
• .github/workflows/lint.yml: registered the new bats file.

How this maps to the 3 failing cases

• triage-auth-token → HIGH (criterion-1 auth area) ✓
• triage-db-migration → HIGH (criterion-1 migration/schema) ✓
• triage-clean-logic-fix → MEDIUM (clean non-trivial logic, no escalate) ✓

The rubric also holds the previously-passing bands: unresolved-thread stays MEDIUM (process escalation, not dangerous content) and docs-typo/test-only stay LOW.

Notes

• The bats guard is the offline-runnable test. The 5/5 held-out acceptance criterion is a behavioral check that requires the live Haiku-tier model via the skill-eval-report workflow (workflow_dispatch, skill=triage) — it cannot be exercised offline here, and tracking issue Skill eval health — triage (holdout) #814 auto-closes on the recovered run.
• escalate behavior was already correct and is deliberately unchanged.
• No new files under evals/; held-out and dev case data are byte-for-byte unchanged.