Repository navigation
feat(developer-experience): claude plugin eval suite in GitHub Actions #6649
Description
Refs: #6302
Summary
The developer-experience plugin's first PR measures its skills with a hand-rolled harness: nested claude -p runs in three arms (no plugin, team conventions in AGENTS.md, plugin) on fixture copies outside this repository. This issue tracks the follow-up: a claude plugin eval suite for the plugin that runs in GitHub Actions, so the measurement repeats on every change instead of by hand.
Why it is a separate issue
claude plugin evalrefuses Bash-granting cases on the maintainer's WSL host (socat missing; Docker Desktop's~/.docker/contextssymlink into/mnt/c, which the eval pre-flight refuses; seen on Claude Code 2.1.289): melodic-software/provisioning#741.workflow_dispatchonly triggers a workflow whose file is on the default branch (docs.github.com, "Manually running a workflow"), so the CI job can only run once this lands on main.claude plugin evalloads no project CLAUDE.md,.claude/or.mcp.jsonfrom the workspace (code.claude.com plugin-evals page), so the conventions-only arm stays in the hand-rolled harness; plugin eval covers the with/without-plugin comparison and the presence and firing cases.
Proposed shape
- A
workflow_dispatchjob onubuntu-24.04that installsbubblewrapandsocatand lifts the Ubuntu 24.04 AppArmor user-namespace restriction for bwrap (sysctl kernel.apparmor_restrict_unprivileged_userns=0), the step Anthropic's sandbox-runtime CI runs. MEDIUM confidence; the first dispatch settles it. - One GitHub environment per Claude account, each holding its own
CLAUDE_CODE_OAUTH_TOKENsecret, so eval spend is split from the review account. - First run checks whether
CLAUDE_CODE_OAUTH_TOKENreaches the eval run: the plugin-evals page says most of the shell environment is withheld (an allowlist plusEVAL_*variables), and its CI example names onlyANTHROPIC_API_KEY. - Cases ported from the PR's scenarios (S1-S7, T1-T4, N1, stays-quiet), deterministic graders where possible and
llmgraders checked against labeled samples. - The provisioning#741 eval distro mirrors the job locally for pre-push runs.
Done when
- The workflow runs the suite on a dispatch and posts the with/without delta per case.
- The OAuth-secret question above is answered with the run's evidence.
Activity
github-actions commentedon Oct 9, 2026
Thanks for the report. Automated triage handed this issue to a maintainer.
Automated triage. A maintainer reviews it.
kyle-sexton commentedon Oct 9, 2026
Carry into this suite: in the first PR's eval (fourth pass, 2026-10-09),
build-clidescribed a default for an unanswered open choice in 1 of 5 plan runs (S4 #10, the withhold rule), and two pass/fail checks were unreachable under that harness's frozen allowlist (rmfor setup's migrate, compound shell commands for the tracker case). A suite with its own permission set should measure all three directly.kyle-sexton commentedon Oct 9, 2026
Also carry: the T4 #2 judge item (setup reports conventions found in the old
.claude/location) did not calibrate (Sonnet TPR 0.70) because 3 model-produced labels read the rubric literally. Before porting that case, have a person relabel those 3 samples, then re-calibrate.kyle-sexton commentedon Oct 11, 2026
The credential question here changes: Max and Team plans now carry monthly API credits in the linked Console org (help article 17154008). Credits cover API-key traffic only; runs started by the Claude Code GitHub Action are excluded even with
-p. Whether a self-writtenclaude -porclaude plugin evalstep with an org key draws credits is the probe in #6962. The credential choice for CI lanes is tracked on the capacity map #6958 (#6968), and ADR 0037 is being amended to allow the operator's own org keys (#6961).kyle-sexton commentedon Oct 13, 2026
Decision recorded. The owner accepted the adjudicated recommendation on 2026-10-12.
Build now: port the developer-experience cases (S1-S7, T1-T4, N1, stays-quiet; T4 #2 held until a person relabels its 3 samples) into plugins/developer-experience/evals in the case.yaml/graders format, and run them on the existing eval-browser-dispatch.yml lane with its shared CLAUDE_CODE_OAUTH_TOKEN as the stated interim; the per-account split or linked-org key waits on #6968/#6962. Route: wayfind-blocked -> ready to implement.
Why: The seed's blocker is false: origin/main 736fb71 has .github/workflows/eval-browser-dispatch.yml (workflow_dispatch, free-form target, allow-tools input, secrets.CLAUDE_CODE_OAUTH_TOKEN at line 203, --max-cost-usd 5), and the owner himself dispatched run 38153781700 on 2026-10-11T16:02 which ran both arms with model turns on CLI 2.1.296, so the issue's OAuth-reachability question is answered and eval spend already draws on the shared token by the owner's own act. The issue body's per-account environment proposal (2026-10-09) is superseded by the owner's newer 2026-10-11T03:52 comment moving the credential choice to #6968 under map #6958, so rule 2 leaves no standing owner decision against the interim and nothing for the owner to decide here beyond what #6968 already holds. ADR 0062 (accepted 2026-10-11) decides only that keys from linked orgs may fund claude -p work and defers storage and lane selection to #6958, and probe #6962 names this issue, so the key route is not available yet.
Precedence applied: Rule 2 decided the one disagreement (A: owner_must_decide on interim shared-token spend; B: proceed on the existing token). The owner's 2026-10-11T03:52 comment is the newest dated decision on the issue and moves the credential choice to #6968, superseding the 2026-10-09 body proposal; the owner's own dispatch of run 38153781700 on the shared token at 16:02 the same day confirms the course. Rules 3 and 4 (ADR 0062, plugin-evals and authentication pages) agree and add nothing that conflicts.
Sources:
Next: (1) Add plugins/developer-experience/evals//{prompt.md,case.yaml,graders/*.md} ported from the #6302 harness scenarios, deterministic graders where possible, llm graders calibrated against labeled samples, carrying the S4 #10 withhold rule and the rm and compound-shell checks; leave T4 #2 out until a person relabels its 3 samples. Validate with plugins/evals/skills/validate/scripts/validate-cases.py; open as a draft PR. (2) After merge, dispatch eval-browser-dispatch.yml with target=plugins/developer-experience, tag empty, allow-tools covering the rm and compound-command cases, env-scrub left on; if the CLI cost preflight exceeds the hard-coded --max-cost-usd 5, raise it in the lane in the same or a follow-up PR. (3) Comment on #6649 that the OAuth-secret done-criterion is answered by run 38153781700 and that the per-account split is tracked on #6968. Do not add a second eval workflow or per-account environments now, and never set ANTHROPIC_API_KEY beside the OAuth token in the job (the key takes precedence and changes which account pays). (4) Open: whether the DX cases need tools beyond the lane's install, and where the #6302 fixtures live; the first dispatch and the porter settle these.
Researched by two independent agents (local evidence and upstream docs) and adjudicated by a third.
kyle-sexton commentedon Oct 13, 2026
Owner batch-ratified this admission in an attended session on 2026-10-12: work-class: scoped. Rule: class C1-C3, no trust surface, the independent check confirmed the recorded decision, and no owner step remains.