[Phase 1] Make the Skill Eval Report visible — surface regression/error outcomes (no longer silent)

[don-petry]

Part of epic #581 (self-improving skills). Follow-up to #584 / #741.

Why

skill-eval-report.yml (#584) runs the deterministic holdout scorer daily (07:00 UTC) and on dispatch, but its result is effectively invisible:

• It is intentionally non-blocking (continue-on-error: true) — a held-out regression or a scorer hard-error never fails the workflow.
• It has permissions: contents: read only, so it writes no issue and no PR comment. Output lives solely in the workflow run's step summary, an artifact (eval-report.json), and a ::notice::/::warning:: annotation.

Net effect: a real skill regression (outcome=regression) — or, worse, a blind scorer failure (outcome=error, hidden because the workflow still goes green) — is only discoverable by manually opening the Actions run. Continuous skill-quality "visibility" that nobody sees defeats the purpose of Phase 1.

What

Surface the daily eval outcome where a human will actually see it, without making it blocking:

• On outcome=regression or outcome=error: open or update a single tracking issue (idempotent — update the same issue in place, never spam a new one per day), labelled e.g. eval-health, containing the score table + regressed-case detail and a link to the run.
• On recovery (outcome=pass after a prior open tracking issue): comment + close the tracking issue.
• Alternative/complementary: fold the latest score + outcome into the existing daily-pr-review-health digest so the org has one daily health surface instead of two silent crons.

Acceptance criteria

• A regression or error in the scheduled run produces a visible, de-duplicated notification (tracking issue or digest line) — not just an Actions annotation.
• Healthy (pass) runs create no noise (no new issue/comment when nothing is wrong).
• The workflow stays non-blocking — continue-on-error preserved; this story adds visibility only, no gating.
• Least-privilege: add only the issues: write scope required for the notify step; keep contents: read for checkout.
• An offline test (stub, no network) covers the open/update/close + de-dupe logic, wired into lint.yml (mirrors the #583/#584 offline-test pattern).

Out of scope

• Making the eval blocking / gating merges (a later phase decision).
• Growing the holdout case set or fixing per-trigger coverage (separate follow-up — note workflow-secrets and unresolved-threads currently live only in dev/, so they are not scored).
• Reducing scorer/model nondeterminism (noise-floor) — separate.

Filed from a monitoring review after #719/#721/#741 landed.

[github-actions]

Thanks for the thorough write-up and for connecting this back to the epic — the visibility gap is well-articulated! Before we start on implementation: for the tracking-issue approach, should the issue be opened in this same repo (petry-projects/.github-private) or in a separate ops/observability repo, and is there an existing eval-health label to reuse or should the workflow create it on first use?

[don-petry]

Dev-Lead Implementation Plan

Issue: #747 — [Phase 1] Make the Skill Eval Report visible — surface regression/error outcomes (no longer silent)

Scope

Add a non-blocking visibility layer to skill-eval-report.yml: on outcome=regression|error, open-or-update a single de-duplicated eval-health tracking issue; on recovery (pass with a prior open issue), comment + close it. Healthy runs stay silent. Logic lives in a stub-testable script with offline bats coverage.

Implementation Checklist

• scripts/evals/notify-eval-health.sh — new: find-or-create/update/close the single eval-health tracking issue (idempotent, de-duped by stable title+label), rendering the score table + regressed-case detail + run link. Modeled on scripts/idea-triage/upsert-queue.sh. Honors DRY_RUN; uses gh from PATH (stub-injectable).
• .github/workflows/skill-eval-report.yml — add issues: write to the job permissions (keep contents: read); add an if: always() "Notify eval health" step that invokes the script with OUTCOME / SKILL / REPORT_PATH / RUN_URL. continue-on-error on the eval step is preserved (no gating added).
• tests/test_eval_health_notify.bats — new offline test (stub gh on PATH, no network) covering open / update / close / de-dupe / no-noise-on-pass.
• .github/workflows/lint.yml — add the new bats file to the repo-specific test list.
• tests/test_skill_eval_report.bats — extend with workflow-YAML assertions for the new issues: write scope and the notify step.

Tests to Write

• regression with no existing issue → creates exactly one issue (label eval-health)
• regression with an existing open issue → edits in place, creates none (de-dupe)
• error outcome → opens/updates the tracking issue (not silent)
• pass with a prior open issue → comments + closes it
• pass with no existing issue → no create, no comment (zero noise)
• body contains score table + regressed-case detail + run link
• workflow YAML declares issues: write and wires the notify step

Test Command

bats tests/test_eval_health_notify.bats tests/test_skill_eval_report.bats tests/test_skill_evals.bats (full suite: the bats list in .github/workflows/lint.yml)

[don-petry]

Confirming the defaults so you're unblocked (matches your 00:29 plan):

• Same repo — open the tracking issue here in petry-projects/.github-private, next to the workflow and CODEOWNERS. No separate ops/observability repo.
• Label — there is no existing eval-health label; have the workflow create-it-on-first-use (idempotent find-or-create) so there's no manual setup step. Single issue, de-duped by stable title + the eval-health label, updated in place.

Proceed on that basis. One real-world note to fold in: the first baseline dispatch of skill-eval-report.yml just came back outcome=regression (run 27516878017) — so this notify path will light up immediately on the first scheduled run. Please make sure the issue body renders the per-case expected-vs-actual detail for regressed cases (your checklist covers it) — that's the first thing we'll need to read.

---

Generated by Claude Code

[don-petry]

Dev-Lead: Implementation Complete

Plan Execution

• New scripts/evals/notify-eval-health.sh — idempotent open/update/close of a single de-duplicated eval-health tracking issue (modeled on idea-triage/upsert-queue.sh)
• .github/workflows/skill-eval-report.yml — added issues: write (kept contents: read) and an if: always() "Notify eval health" step; continue-on-error on the eval step preserved (no gating)
• tests/test_eval_health_notify.bats — offline stub-gh tests for open / update / close / de-dupe / no-noise + guards
• tests/test_skill_eval_report.bats — extended with workflow-YAML assertions for the new scope + notify step
• .github/workflows/lint.yml — added the new bats file to the repo-specific test list

Behavior (visibility only, non-blocking)

• outcome=regression|error → opens or updates in place a single eval-health tracking issue (score table + regressed-case detail + run link). Never spams a new issue per day.
• outcome=pass with a prior open tracking issue → comments Recovered and closes it.
• outcome=pass with no open issue → zero noise (no create/comment/close).
• Least-privilege: only issues: write added; contents: read retained for checkout.

Test Results

# bats (full lint.yml list, 23 files): 376 passing, 0 failing
ok 366 regression with no existing issue creates exactly one eval-health issue
ok 367 regression with an existing open issue updates in place (no new issue — de-dupe)
ok 368 the tracking issue is looked up de-duped by label and open state
ok 369 regression body carries the score table, regressed case detail and run link
ok 370 error outcome opens the tracking issue even with an unparseable report
ok 371 error outcome with an existing issue updates it (de-dupe, not silent)
ok 372 pass with a prior open issue comments and closes it (recovery)
ok 373 pass with no existing issue makes zero noise (no create/comment/close)
ok 374 missing REPO is a hard error
ok 375 an unknown OUTCOME is a hard error
ok 376 DRY_RUN never invokes gh mutations

# .dev-lead/scripts/dev-lead-lint.sh
  [lint] shellcheck on 55 file(s)...
  [lint] validating 4 agent profile(s)...
  [lint] all checks passed

Files Changed

• scripts/evals/notify-eval-health.sh (new): notifier logic — find-or-create/update/close.
• .github/workflows/skill-eval-report.yml: issues: write + notify step (always()).
• tests/test_eval_health_notify.bats (new): offline notifier tests.
• tests/test_skill_eval_report.bats: +4 YAML assertions for [Phase 1] Make the Skill Eval Report visible — surface regression/error outcomes (no longer silent) #747.
• .github/workflows/lint.yml: registered new bats file.

Notes

• Chose the tracking-issue approach (primary AC path); the daily-pr-review-health digest fold-in was explicitly "alternative/complementary" and is left out to keep scope minimal.
• De-dup is scoped to open issues by label + stable per-skill title. A human-closed issue is intentionally not reopened; a fresh regression then opens a new one.
• Acknowledged out-of-scope items untouched: no gating/blocking, no holdout case-set growth, no nondeterminism work.