feat: implement issue #691 — [Guardrail] Held-out hygiene: dev/test split so the proposer never scores against cases it can see

[don-petry]

Closes #691

Implemented by dev-lead agent. Please review.

Summary by CodeRabbit

• New Features
  • Added an automated validator for evaluation case JSONL files to enforce required ids, uniqueness, and disjoint dev/holdout splits.
• Tests
  • Extended CI Bats coverage to run the new evaluation-case validation checks.
• Documentation
  • Documented the evals/ directory structure, required case format, dev vs holdout split rules, de-identification expectations, and held-out integrity protections.
• Chores
  • Updated CODEOWNERS to lock ownership for /evals/**/holdout/ paths.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[coderabbitai]

Warning

Review limit reached

@don-petry, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 43 minutes and 23 seconds. Learn how PR review limits work.

Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file).

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 0e91e358-a70e-45e6-8910-ec9b99a2490b

📥 Commits

Reviewing files that changed from the base of the PR and between d999094 and 4d4725e.

📒 Files selected for processing (7)

• .github/CODEOWNERS
• .github/workflows/lint.yml
• evals/README.md
• evals/example-skill/dev/cases.jsonl
• evals/example-skill/holdout/cases.jsonl
• evals/validate-cases.py
• tests/test_validate_cases.bats

📝 Walkthrough

Walkthrough

Introduces a dev/holdout eval split for the self-improving-skills pipeline. A new Python validator (evals/validate-cases.py) discovers skill directories, parses cases.jsonl files, enforces non-empty unique IDs within each split, and rejects any cross-split ID overlap. Example cases are added for both splits, a Bats test suite covers all validator failure modes, CI is updated to run it, holdout paths are CODEOWNERS-gated, and evals/README.md documents the full contract.

Changes

Eval dev/holdout split hygiene

Layer / File(s)	Summary
validate-cases.py validator logic evals/validate-cases.py	Adds the full validator script: fail(), load_split_ids() (per-file ID parsing and uniqueness), validate_skill() (cross-split disjointness check), discover_skills() (directory traversal), and main() entry point with optional CLI path argument.
Example skill cases evals/example-skill/dev/cases.jsonl, evals/example-skill/holdout/cases.jsonl	Adds three JSONL records to the dev split (shellcheck quoting, SHA-pinning actions, tools frontmatter) and three to the holdout split (workflow timeouts, token handling, bash strict mode), providing a concrete exercisable example tree.
Bats tests and CI wiring tests/test_validate_cases.bats, .github/workflows/lint.yml	Adds a ten-case Bats suite covering the happy path and all validator failure modes (cross-split overlap, intra-split duplicate, malformed JSON, missing/empty id, non-object value, blank-line skipping, missing holdout, real-tree smoke). Extends the bats CI job to run the new test file.
CODEOWNERS gate and README .github/CODEOWNERS, evals/README.md	Adds a CODEOWNERS rule locking /evals/**/holdout/ to @petry-projects/org-leads. Adds evals/README.md documenting the dev/holdout split contract, ID discipline, no-overlap rule, de-identification requirement, CODEOWNERS gate rationale, and consumer-side isolation notes.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

• petry-projects/.github-private#83: Modifies .github/CODEOWNERS to assign @petry-projects/org-leads ownership, directly related to this PR's addition of the /evals/**/holdout/ ownership rule to the same file.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 57.14% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title directly references issue #691 and accurately summarizes the main change: implementing a dev/holdout split for evaluation cases.
Linked Issues check	✅ Passed	The PR implements all five acceptance criteria from issue #691: partitions cases into dev/holdout splits, documents the split mechanism, validates uniqueness across splits, applies CODEOWNER gating, and provides comprehensive README documentation.
Out of Scope Changes check	✅ Passed	All changes are directly within scope of issue #691: CODEOWNERS gating, validation script, test suite, documentation, and example eval cases. No unrelated modifications detected.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch dev-lead/issue-691-20260614-1229

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[don-petry]

Dev-Lead — review-changes (no-changes)

No changes were needed for this PR.

[gemini-code-assist]

Code Review

This pull request establishes a structured directory and validation pipeline for held-out evaluation cases under evals/, featuring a Python validator script, comprehensive documentation, example cases, and BATS tests. The review feedback suggests two improvements to the validator script: explicitly specifying UTF-8 encoding when reading files to prevent platform-dependent decoding errors, and outputting error messages to sys.stderr instead of sys.stdout to adhere to standard CLI conventions.

[don-petry]

Dev-Lead — fix-reviews (applied)

Changes committed and pushed.

[don-petry]

Dev-Lead — waiting on PR blockers (intent: review-changes)

PR: #712
No changes were committed, but the PR still has blocking checks or reviews (failing or cancelled checks, or changes-requested reviews). The retry cron will re-attempt automatically. Next attempt after: 2026-06-14T13:16:21Z

[don-petry]

Note

@don-petry I reviewed this PR and no code changes were needed, but it still has blocking checks or reviews (failing or cancelled checks, or changes-requested reviews), so I cannot mark it done yet. I'll re-check automatically.
Next attempt after: 2026-06-14T13:16:21Z

[don-petry]

Dev-Lead — review-changes (no-changes)

No changes were needed for this PR.

[don-petry]

Dev-Lead — waiting on PR blockers (intent: review-changes)

PR: #712
No changes were committed, but the PR still has blocking checks or reviews (failing or cancelled checks, or changes-requested reviews). The retry cron will re-attempt automatically. Next attempt after: 2026-06-14T14:59:42Z

[don-petry]

Note

@don-petry I reviewed this PR and no code changes were needed, but it still has blocking checks or reviews (failing or cancelled checks, or changes-requested reviews), so I cannot mark it done yet. I'll re-check automatically.
Next attempt after: 2026-06-14T14:59:42Z

[don-petry]

Dev-Lead — fix-reviews (no-changes)

Agent reasoning

Addressed 0 threads:
(no open threads to address)
Test verification: skipped — no code changes were made; CI suite is already green
Files changed: none
```
The PR has zero open review threads, all CI checks are passing (two jobs are still `in_progress` but not failing), and no reviewer has submitted a `CHANGES_REQUESTED` review. No action is required.

[don-petry]

Dev-Lead — review-changes (no-changes)

No changes were needed for this PR.

[don-petry]

Dev-Lead — waiting on PR blockers (intent: review-changes)

PR: #712
No changes were committed, but the PR still has blocking checks or reviews (failing or cancelled checks, or changes-requested reviews). The retry cron will re-attempt automatically. Next attempt after: 2026-06-14T21:11:14Z

[don-petry]

Note

@don-petry I reviewed this PR and no code changes were needed, but it still has blocking checks or reviews (failing or cancelled checks, or changes-requested reviews), so I cannot mark it done yet. I'll re-check automatically.
Next attempt after: 2026-06-14T21:11:14Z