feat: implement issue #582 — [Phase 1] Define held-out eval-case format + seed prompts/triage.md case set

[don-petry]

Closes #582

Implemented by dev-lead agent. Please review.

Summary by CodeRabbit

• New Features

  • Established evaluation case validation framework with JSON schema and validator
  • Added 8 new triage evaluation test cases covering various scenarios

• Documentation

  • Added comprehensive guide for evaluation cases including structure, format, and validation requirements

• Tests

  • Added test suite validating evaluation cases against schema and enforcing uniqueness constraints

• Chores

  • Updated CI workflow to include evaluation case validation in automated checks
  • Added code ownership rules to restrict evaluation case modifications

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[coderabbitai]

Warning

Review limit reached

@don-petry, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 59 minutes and 51 seconds. Learn how PR review limits work.

Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file).

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: d984fde5-25e6-46f8-ad25-09f451a4e5d8

📥 Commits

Reviewing files that changed from the base of the PR and between 965ca88 and fe72dec.

📒 Files selected for processing (7)

• .github/CODEOWNERS
• .github/workflows/lint.yml
• evals/README.md
• evals/case.schema.json
• evals/triage/cases.jsonl
• evals/validate-cases.py
• tests/test_evals_cases.bats

📝 Walkthrough

Walkthrough

This PR introduces a held-out evaluation case infrastructure for measuring triage skill quality. It adds a JSON Schema for eval cases, a set of seed reference cases grounded in triage decision criteria, a Python validator script for CI-based validation, comprehensive test coverage, and workflow integration with CODEOWNERS protection against proposer edits.

Changes

Held-out evaluation case infrastructure for triage skill

Layer / File(s)	Summary
Eval case schema and format contract skills/evals/case.schema.json, skills/evals/README.md, .github/CODEOWNERS	JSON Schema defines required triage eval case fields (id, input, expected.escalate, expected.risk) with enum validation and a HIGH-risk-to-escalate coupling rule. README documents case format, validation workflow, and CODEOWNERS guards against proposer edits to prevent reward hacking.
Seed evaluation cases covering triage scenarios skills/evals/triage/cases.jsonl	Eight JSONL reference cases span safe low/medium-risk approvals (docs typos, test-only changes, logic fixes) and escalations for high-risk triggers (auth/tokens, DB migrations, SQL injection, workflow secrets) plus unresolved threads.
Validator script for case validation skills/evals/validate-cases.py	Python validator loads the JSON Schema, reads JSONL cases with blank-line tolerance, validates each case against the schema with precise error reporting, enforces cross-line case-id uniqueness, and exits with GitHub Actions error formatting or a success summary.
Comprehensive validator test suite tests/test_evals_cases.bats	Bats test suite verifies seed cases pass validation, rejects cases with missing/invalid fields and schema violations, enforces case-id uniqueness and the HIGH-risk-to-escalate invariant, validates schema structure matches Draft2020-12, and confirms seed data covers both escalation outcomes and required high-risk tags.
Workflow integration and test automation .github/workflows/lint.yml	Workflow now triggers on skills/** path changes and includes tests/test_evals_cases.bats in the CI test suite.

🎯 2 (Simple) | ⏱️ ~12 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title directly references issue #582 and accurately summarizes the primary change: defining a held-out eval-case format and seeding the triage case set with prompts grounded in triage.md.
Linked Issues check	✅ Passed	All four acceptance criteria from issue #582 are satisfied: cases.jsonl with escalate/risk fields [#582-AC1], schema + README with jsonschema validation [#582-AC2], cases cover both escalate values and required high-risk triggers [#582-AC3], and CODEOWNERS entry added [#582-AC4].
Out of Scope Changes check	✅ Passed	All changes are tightly scoped to issue #582 Phase 1 objectives: eval-case format definition (schema, README), seed cases, validation script, tests, and CI workflow updates. No unrelated modifications detected.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch dev-lead/issue-582-20260612-0136

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[don-petry]

Dev-Lead — review-changes (no-changes)

No changes were needed for this PR.

[gemini-code-assist]

Code Review

This pull request introduces a framework for held-out evaluation cases to measure prompt-skill quality, including a JSON schema, seed triage cases, a Python validation script, and BATS tests. The feedback suggests strengthening the JSON schema by adding conditional validation to enforce that high-risk cases must escalate, ensuring platform-independent file reading in the validator by specifying UTF-8 encoding, and adding a test to verify the new schema constraint.

[don-petry]

Dev-Lead — review-changes (applied)

Changes committed and pushed.

[don-petry]

Auto-rebase failed — merge conflict — this branch has conflicts with main that must be resolved.

Claude will attempt to resolve this automatically. If it cannot, a follow-up comment will explain what needs manual attention.

To resolve manually instead:

git fetch origin
git merge origin/main
# resolve conflicts, then:
git add .
git commit
git push

[donpetry-bot]

Superseded by automated re-review at f342a2e52599db95e12f2ae8083b66be27ccb854 — click to expand prior review.

Review — fix requested (cycle 1/3)

The automated review identified the following issues. Please address each one:

Findings to fix

Automated review — NEEDS HUMAN REVIEW

Risk: LOW
Reviewed commit: eeeb514eb33d51f1faef6989a32f91797de1602d
Cascade: triage → deep (triage: haiku 4.5 → deep: opus 4.8 + duck: o4-mini → audit: fable 5)

Summary

Phase 1 held-out eval-case infrastructure for issue #582 (schema, validator, seed triage cases, README, CODEOWNERS guard, bats tests). Code is low-risk and well-structured, CI is green, and the advisory-bot suggestions (schema if/then HIGH->escalate constraint, HIGH+escalate=false rejection test, UTF-8 reads) are already implemented in the diff. The PR cannot be approved only because it is not mergeable: mergeable=CONFLICTING / mergeStateStatus=DIRTY after auto-rebase failed against main.

Findings

• MAJOR: PR is CONFLICTING (mergeStateStatus DIRTY). Auto-rebase failed with a merge conflict against main (see auto-rebase-conflict comment). Branch must be rebased/merged and conflicts resolved before this can be approved/merged. This is the single blocking gate.
• INFO: All four acceptance criteria of [Phase 1] Define held-out eval-case format + seed prompts/triage.md case set #582 are satisfied: cases.jsonl covers both escalate=true/false and includes auth-secrets, db-migration, and security-anti-pattern triggers (AC#1,Add @claude delegation, auto-merge, and rebase handling #3); case.schema.json + README document the format and validate via python jsonschema draft 2020-12 like plan.schema.json (AC#2); CODEOWNERS gates /skills/evals/ separately as the reward-hacking guard (AC#4).
• INFO: Advisory bot findings are already resolved in the diff: Gemini's if/then schema constraint enforcing HIGH risk pairs with escalate=true is present (case.schema.json), the suggested validate-rejects-HIGH-with-escalate-false test is present (tests/test_evals_cases.bats), and validate-cases.py already reads files with encoding=utf-8. SonarCloud and CodeRabbit passed. (skills/evals/case.schema.json line 56)
• INFO: No security concerns: validate-cases.py is a pure structural validator (jsonschema only, no eval/exec/shell/network). The CODEOWNERS change tightens ownership of the held-out set (a deliberate reward-hacking guard), it does not loosen review requirements. The lint.yml change only adds a path filter and a new bats test file; no new workflow triggers, secrets, or pull_request_target usage. (.github/CODEOWNERS)

---

Reviewed by the PR-review cascade (triage: haiku 4.5 → deep: opus 4.8 + duck: o4-mini → audit: fable 5). Reply if you need a human review.

Additional tasks

1. Resolve all unresolved review thread comments from other reviewers
2. Ensure all CI checks pass after your changes
3. Rebase on the target branch if behind
4. Do NOT modify files unrelated to the findings above

The review cascade will automatically re-review after new commits are pushed.

[don-petry]

Auto-rebase failed — merge conflict — this branch has conflicts with main that must be resolved.

Claude will attempt to resolve this automatically. If it cannot, a follow-up comment will explain what needs manual attention.

To resolve manually instead:

git fetch origin
git merge origin/main
# resolve conflicts, then:
git add .
git commit
git push

[don-petry]

Dev-Lead — rebase (no-changes)

Agent reasoning

PR: #623
Rebased onto: main
Conflicts resolved: 1 file
- .github/workflows/lint.yml: Merged bats test list — kept main's test_oversized_pr.bats,
  dropped test_initiative_planner_redispatch.bats (not in main), added PR's test_evals_cases.bats
Push: success
```
**Notes on the unusual history:** The PR branch had 383 commits with no common ancestor with main (main was squashed to a single commit). A standard `git rebase` would have replayed all 383 commits, causing widespread conflicts. The resolution was to cherry-pick only the two PR-specific commits (`c680e9a` feat: #582, `734bb1d` chore: manual instructions) onto a fresh branch from main.

[don-petry]

Dev-Lead — fix-bot-comment (no-changes)

Agent reasoning

Issues addressed: 0
Quality Gate Status: Passed
Files changed: No fixes required
Skipped (no actionable issues): All
```

[don-petry]

Dev-Lead — waiting on PR blockers (intent: review-changes)

PR: #623
No changes were committed, but the PR still has blocking checks or reviews (failing or cancelled checks, or changes-requested reviews). The retry cron will re-attempt automatically. Next attempt after: 2026-06-13T13:18:53Z

[don-petry]

Note

@don-petry I reviewed this PR and no code changes were needed, but it still has blocking checks or reviews (failing or cancelled checks, or changes-requested reviews), so I cannot mark it done yet. I'll re-check automatically.
Next attempt after: 2026-06-13T13:18:53Z

[don-petry]

Dev-Lead — review-changes (no-changes)

No changes were needed for this PR.

[don-petry]

Dev-Lead — review-changes (no-changes)

No changes were needed for this PR.

[don-petry]

Dev-Lead — fix-bot-comment (no-changes)

Agent reasoning

- ✅ Lint: success
- ✅ ShellCheck: success  
- ✅ CodeQL: success
- ✅ Tests (bats, unit): success
- ✅ Agent validation: success
---
**Bot:** sonarqubecloud[bot]
**Issues addressed:** 0 (Quality Gate passed with no findings)
**Files analyzed:** `.github/CODEOWNERS`, `.github/workflows/lint.yml`, `evals/README.md`, `evals/case.schema.json`, `evals/triage/cases.jsonl`, `evals/validate-cases.py`, `tests/test_evals_cases.bats`
**Skipped (informational):** 0

[don-petry]

Dev-Lead — review-changes (no-changes)

No changes were needed for this PR.

[donpetry-bot]

Automated review — APPROVED ✓

Risk: MEDIUM
Reviewed commit: f342a2e52599db95e12f2ae8083b66be27ccb854
Review mode: triage-approved (single reviewer)

Summary

Phase-1 held-out eval-case framework for the triage skill: a draft 2020-12 JSON schema, a structural-only validator (no scorer/model), 8 seed triage cases, BATS tests, a CODEOWNERS guard, and a lint path/test wiring. Small, self-contained, and fully gated by green CI. Triage-tier low-risk assessment confirmed.

Linked issue analysis

Closes #582 and substantively satisfies all four acceptance criteria: (1) evals/triage/cases.jsonl pairs a triage input with expected escalate/risk; (2) case.schema.json + README mirror the scripts/initiative-planner/plan.schema.json python-jsonschema pattern; (3) cases cover both escalate outcomes and include auth-secrets, db-migration, and security-anti-pattern triggers; (4) a CODEOWNERS entry separately owns the held-out set as the reward-hacking guard. Note: the issue named skills/evals/ paths, but this repo has no skills/ tree (the triage skill is prompts/triage.md), so placing the set at top-level evals/ is a correct adaptation rather than a deviation — the README/schema correctly reference prompts/triage.md.

Findings

No blocking findings. All three gemini-code-assist suggestions are present in the final state and their threads are resolved: the schema enforces HIGH => escalate=true via if/then; validate-cases.py reads files with encoding="utf-8"; and a dedicated test rejects HIGH-with-escalate-false. The validator is intentionally validator-only (no model invocation), so the suite runs offline. CODEOWNERS and lint.yml edits are minimal, additive, and security-strengthening (no secrets, no pull_request_target, no new permissions). Validator run locally against the seed set: 'cases OK: 8 case(s), unique ids, schema-valid.'

CI status

All required checks green (Lint, shellcheck/ShellCheck, bats, unit-tests, CodeQL Analyze python/actions, SonarCloud, agent-shield/AgentShield, Agent Security Scan, Secret scan gitleaks, review/review, validate-agent-profiles, gh-aw-compile, CodeRabbit). Remaining checks SKIPPED (dependency-audit ecosystems, dependabot-automerge, dev-lead ci-relay) — none failing.

---

Reviewed automatically by the PR-review agent (single-reviewer mode: fable 5). Reply if you need a human review.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[don-petry]

Dev-Lead — waiting on PR blockers (intent: review-changes)

PR: #623
No changes were committed, but the PR still has blocking checks or reviews (failing or cancelled checks, or changes-requested reviews). The retry cron will re-attempt automatically. Next attempt after: 2026-06-14T20:25:12Z

[don-petry]

Note

@don-petry I reviewed this PR and no code changes were needed, but it still has blocking checks or reviews (failing or cancelled checks, or changes-requested reviews), so I cannot mark it done yet. I'll re-check automatically.
Next attempt after: 2026-06-14T20:25:12Z