Claude Code Opus baseline: readiness and full-suite smoke gate #62

This was generated by AI during triage.

Parent

#50 - PRD: Codex and Claude Code starter-suite comparison

Current behavior / design

The project has a completed Claude Code starter-suite baseline using claude-haiku-4-5-20251001, selected because it was the cheap proof-of-concept Claude Code configuration. The first comparison report explicitly calls out that this is asymmetric against Codex gpt-5.5 / xhigh.

The project does not yet have a larger Claude Code baseline, and Opus may be expensive or quota-prone.

Why change

To make later comparisons more apples-to-apples, the project should test a high-capability Claude Code configuration. Because Opus may consume more budget or hit quota, this work should proceed in phases and stop after the first full-suite smoke before repeated collection.

What to build

Prepare and run a Claude Code Opus starter-suite smoke gate:

1. Verify the exact Opus model/configuration available to the local Claude Code account.
2. Record a readiness/config note under an appropriate reports/claude-code-opus-.../ bundle.
3. Run one sequential Claude Code smoke trial for each of the 12 starter-suite tasks using one trial and one job.
4. Stop after those 12 smoke trials.
5. Inspect cost, quota behavior, validity, failures, and suspicious/resource-heavy runs.
6. Comment a recommendation about whether to collect the remaining four fair trials per task in a follow-up issue.

Do not collect the remaining repeated trials in this issue unless Jordan explicitly approves after reviewing the smoke evidence and cost/quota result.

Acceptance criteria

• The exact Opus model request and event-derived runtime model identity are recorded.
• Claude Code CLI version, account/auth shape, permission mode, output format, timeout, max-turns decision, session persistence, and effort setting if available are recorded.
• A readiness/config note is committed under a report bundle.
• One sequential smoke trial is run for each starter-suite task with --trials 1 --jobs 1 semantics.
• Each smoke trial is marked valid or excluded with explicit validity and exclusion metadata.
• Fair failures are retained as capability evidence rather than rerun away.
• Cost, quota behavior, median duration, and failure patterns are summarized in an issue comment or committed smoke review packet.
• The issue stops after the 12-trial smoke unless Jordan approves repeated collection.
• Any required adapter/config changes are tested before merge.

Stop conditions

• Stop if the local account cannot access the intended Opus model.
• Stop if Claude Code falls back to an unexpected model and the runtime model identity cannot be trusted.
• Stop if quota, account limits, or cost make repeated collection irresponsible.
• Stop if any task failure appears to be caused by an unfair task, grader, setup, dependency, operator, or eval-harness issue rather than the agent harness.

Out of scope

• Running the full five-trial-per-task Opus baseline without a post-smoke approval step.
• Changing starter-suite prompts or graders to help Opus.
• Writing the final multi-config comparison report.