[Phase 1] Wire the eval scorer into CI as a non-blocking report

[github-actions]

Story

As a skill maintainer,
I want a scheduled/dispatch CI workflow that runs the scorer on the held-out set and publishes a non-blocking score report,
so that skill quality is continuously visible without gating any merge — completing the idea's Deliverable 1.

Acceptance Criteria

1. A workflow runs run-eval.sh triage on a cron schedule and on workflow_dispatch and publishes the score report (artifact and/or issue/PR comment).
2. The job is report-only: a regression does NOT fail the workflow or block any PR (explicit continue-on-error / non-blocking step).
3. The real-model eval runs only on the scheduled/dispatch path — never on every PR — to control model cost; PR-time validation is limited to the offline tests from story 2.
4. Third-party actions are SHA-pinned per AGENTS.md and the workflow declares least-privilege top-level permissions.

Tasks / Subtasks

• Add a scheduled+dispatch workflow modeled on daily-pr-review-health.yml (AC: test issue from agent #1, Add @claude delegation, auto-merge, and rebase handling #3)
  • cron + workflow_dispatch triggers
  • optional lookback/skill input
• Invoke run-eval.sh and upload/comment the report as non-blocking (AC: test issue from agent #1, Go-live improvements for PR review agent #2)
• SHA-pin actions and set permissions: minimal (AC: Optimize review: small-PR and incremental fast paths #4)

Dev Notes

• Model the workflow on .github/workflows/daily-pr-review-health.yml (scheduled cron 0 6 * * *, lookback input, read-mostly) — it is the closest existing scheduled health-report pattern.
• Non-blocking is the entire point: the idea specifies 'wired into CI as a non-blocking report (no loop yet)'. Use continue-on-error or a report-only step; never let an eval regression fail CI or gate a merge in Phase 1.
• Do not run the model-backed eval on pull_request events — it costs real tokens per case. Keep PR-time coverage to the offline bats tests; reserve the live eval for cron/dispatch.
• SHA-pin third-party actions (the mutable channel-tag exception in AGENTS.md applies only to first-party @/ refs, not to third-party actions).

Project Structure Notes

Adds one workflow under .github/workflows/ in the org-private-automation class (like initiative-driver.yml), not a thin caller stub.

References

• .github/workflows/daily-pr-review-health.yml
• .github/workflows/initiative-planner.yml
• skills/evals/run-eval.sh
• AGENTS.md

Likely target surface

• .github/workflows/skill-eval-report.yml

Story prepared by the BMAD Scrum Master (Bob) for epic #581. Status: ready-for-dev.

[don-petry]

Dev-Lead Implementation Plan

Issue: #584 — [Phase 1] Wire the eval scorer into CI as a non-blocking report

Scope

Add a scheduled + workflow_dispatch workflow .github/workflows/skill-eval-report.yml that runs scripts/evals/run-eval.sh <skill> against the held-out set and publishes a non-blocking score report (JSON artifact + step summary) — never gating a merge and never running on pull_request events.

Implementation Checklist

• .github/workflows/skill-eval-report.yml — new org-private-automation workflow modeled on daily-pr-review-health.yml:
  • cron schedule + workflow_dispatch with optional skill input (default triage)
  • no pull_request trigger (keep the real-model eval off PR-time — AC Add @claude delegation, auto-merge, and rebase handling #3)
  • eval step is report-only via continue-on-error: true so a regression (scorer exit 1) never fails the workflow (AC Go-live improvements for PR review agent #2)
  • publish report as an uploaded artifact + $GITHUB_STEP_SUMMARY (AC test issue from agent #1)
  • third-party actions SHA-pinned; top-level permissions: {} with a least-privilege job block (AC Optimize review: small-PR and incremental fast paths #4)
• Register the new bats test in .github/workflows/lint.yml's bats list (per AGENTS.md)

Tests to Write

• tests/test_skill_eval_report.bats — parses the workflow YAML and asserts:
  • schedule (cron) and workflow_dispatch triggers present; no pull_request trigger
  • invokes scripts/evals/run-eval.sh
  • eval step sets continue-on-error: true (non-blocking)
  • top-level permissions declared and least-privilege (job grants only what it needs)
  • every uses: action is SHA-pinned (40-hex)
  • optional skill dispatch input present

Test Command

bats tests/test_skill_eval_report.bats (and the full bats suite + bash .dev-lead/scripts/dev-lead-lint.sh)

[don-petry]

Dev-Lead: Implementation Complete

Plan Execution

• Added .github/workflows/skill-eval-report.yml — scheduled (cron: 0 7 * * *) + workflow_dispatch (optional skill input, default triage); no pull_request trigger
• Eval step invokes scripts/evals/run-eval.sh "$SKILL" and is report-only via continue-on-error: true — a scorer regression (exit 1) never fails the workflow or gates a merge
• Publishes the report as an uploaded artifact + a rendered $GITHUB_STEP_SUMMARY table (with regressed-case detail)
• Third-party actions SHA-pinned (checkout, setup-node, cache, upload-artifact); top-level permissions: {} with a least-privilege contents: read job block
• Added tests/test_skill_eval_report.bats (12 assertions over AC#1–Optimize review: small-PR and incremental fast paths #4) and registered it in lint.yml's bats list

Acceptance Criteria Mapping

• AC#1 — cron + dispatch run the scorer and publish the report (artifact + step summary). ✔
• AC#2 — explicit continue-on-error: true on the eval step keeps a regression non-blocking. ✔
• AC#3 — live model eval runs only on schedule/dispatch; no pull_request/pull_request_target trigger (PR-time coverage stays the offline tests/test_skill_evals.bats). ✔
• AC#4 — all actions SHA-pinned; least-privilege top-level permissions: {}. ✔

Test Results

# New workflow-contract test (red → green):
1..12
ok 1 workflow file exists
ok 2 parses as a YAML mapping
ok 3 AC#1/#3: triggers on a cron schedule
ok 4 AC#1/#3: triggers on workflow_dispatch
ok 5 AC#3: never triggers on pull_request (no PR-time model eval)
ok 6 AC#1: exposes an optional skill dispatch input
ok 7 AC#1: invokes scripts/evals/run-eval.sh
ok 8 AC#1: publishes the report (artifact upload or step summary)
ok 9 AC#2: the eval step is non-blocking (continue-on-error: true)
ok 10 AC#4: declares a top-level permissions block
ok 11 AC#4: top-level permissions are least-privilege (not write-all)
ok 12 AC#4: every third-party action is SHA-pinned (40-hex)

# Full lint.yml bats suite: 323/323 ok (tests 312–323 are the new file).
# bash .dev-lead/scripts/dev-lead-lint.sh: shellcheck (53 files) + agent
#   profiles (4) — all checks passed.
# Both edited workflows parse as valid YAML.

Files Changed

• .github/workflows/skill-eval-report.yml (new): scheduled/dispatch non-blocking eval-report workflow.
• tests/test_skill_eval_report.bats (new): pins the AC#1–Optimize review: small-PR and incremental fast paths #4 contract by parsing the workflow YAML offline (no live model call).
• .github/workflows/lint.yml: registered the new bats file in the test list (per AGENTS.md's repo-specific-test-list exception).

Notes

• Chose artifact + step summary (not an issue comment) to publish the report — AC#1 allows "artifact and/or comment", and this keeps the job at least-privilege contents: read (no issues: write).
• actionlint is not part of this repo's CI lint set and is not installed in the runner; validation relied on the repo's own bats + shellcheck + PyYAML checks.
• Per dev-lead constraints, no commit/push/PR was created — the CI workflow handles git operations.