Re-run Claude Code evals on a clean channel before re-publishing its report #1

Activity

1. angel291592

   added

   documentationImprovements or additions to documentationImprovements or additions to documentation

   help wantedExtra attention is neededExtra attention is needed

   on Oct 3, 2026
2. angel291592 commented on Oct 3, 2026

   

   angel291592

   on Oct 3, 2026

   OwnerAuthor

   More actions

   Tracked as a real work item, not a stale note. The report stays held back until the clean-channel re-run passes; the acceptance criteria in this issue's checklist are the right ones, so nothing here needs to be softened.

   Current state, for anyone picking this up:

   • The 2026-09-22 Claude Code report was removed and Claude Code sits at spec-compatible; the methodology and cases remain public (CHANGELOG 0.2.0).
   • The unrestricted-channel re-run of the full 10-case suite (TIMEOUT 480s, ungrillable included) is the blocking item — the first two checklist entries.
   • Re-publishing is a three-part change, and all three must land together: drop the evals/reports/*-claude-code.md line from .gitignore, re-commit the report, and restore verified in both READMEs. Restoring verified without the committed report would make the READMEs claim evidence the repo does not carry.

   Acceptance: the suite passes on the clean channel, then the .gitignore line, the report, and both READMEs are updated in the same commit.

   Labeling this documentation rather than bug: the skill's behaviour is not the finding here — the channel was, which is why the report was withdrawn instead of patched. If a re-run on an unrestricted channel still shows the same failures, this becomes a bug and should be relabeled.
3. angel291592 commented on Oct 10, 2026

   

   angel291592

   on Oct 10, 2026

   OwnerAuthor

   More actions

   Status check against the current setup, 2026-10-10.

   The first checkbox does not hold. The only claude-code record on disk from after 2026-09-22 is a five-case subset (question-not-trigger, add-caching-two-turn, git-only-fact, no-repo, chinese-ambiguous, one run each). It was taken with thinking forced off, on the same channel that answered English prompts in Chinese — the condition this issue exists to exclude — and ungrillable is not in it, so the 480 s / include-ungrillable requirement is unmet too. That subset came out 2/5, and because of the channel none of it can be attributed to the skill. If a full unrestricted run exists elsewhere, its record is not on this machine; either way the three-part change below has nothing to attach to yet.

   What has changed since. The harness is now 2.1.284 (it was 2.1.77), and a three-session probe run today — unrestricted default configuration, no thinking override, model left at the harness default — came back clean:

   check	result
   contamination check	OK — English prompt, English answer
   skill visible in a prepared workspace	yes, and the reply was in English
   add-caching-auto	1/1 pass, 0 hallucinated evidence, 0 over-asks, 0 timeouts

   So the channel that made these runs un-attributable looks usable again, and the blocker is no longer obviously infrastructure. What a publishable run still needs, in order:

   1. The full suite — 17 cases now, ungrillable included — with the model named explicitly rather than left at the harness default, so the report can state what actually ran.
   2. On pass, all three of: drop the evals/reports/*-claude-code.md line from .gitignore, commit the report, restore verified in both READMEs — one commit, as recorded above.

   Until (1) exists, nothing here changes: Claude Code stays spec-compatible and the report stays held back. No code change can close this one; only a run can.