Migrate brittle starter graders to hidden verifiers and semantic checks #123

Summary

After hidden verifier patch support exists, audit the starter suite and migrate brittle graders away from source-token checks where the intended requirement can be tested behaviorally or semantically.

Context

Prompt-hint removal exposed false negatives where model patches appeared behaviorally acceptable, but deterministic graders rejected them for using a different source shape than the reference patch.

Examples observed in the starter-coding suite:

• click-should-strip-ansi-tests-001: submitted parametrized tests covered the requested ANSI/color/stream behavior, but the grader required literal source tokens such as color=True, color=False, sys.stdin, sys.stdout, and sys.stderr.
• remotion-audio-context-autoplay-muted-001: submitted code guarded the audio-context resume/wait path with an outer if (!muted), but the grader only recognized two narrower textual shapes.
• click-help-option-refactor-001: submitted code shared help-option construction, but the grader scanned globally for a nested show_help callback string instead of checking whether the duplicate implementation was actually removed.

These examples are motivating cases, not the boundary of the work. The intended scope is a starter-suite-wide audit: any starter task whose current grader is making the benchmark brittle or overly source-shape-specific should be considered for hidden verifier migration or semantic structural checking.

Chosen Design

• Audit all task bundles under tasks/starter/.
• Use hidden verifier patches/tests for behavioral requirements wherever possible.
• Keep deterministic public graders thin for behavior: run public commands, report hidden verifier outcomes, and enforce task/result hygiene.
• Reserve deterministic structural checks for genuinely structural requirements.
• Prefer semantic or AST-style structural inspection over raw string-token checks when structure still matters.
• Leave artifact/change-capture policy, such as ignored/generated __pycache__ handling, out of this issue unless a task-specific grader needs to stop conflating that policy with behavior.
• Start with at least one representative real task migration and validate it with one trial and one job before scaling the same pattern to the rest of the starter-suite audit.

Implementation Boundary

In scope:

• Reviewing every current tasks/starter/* task card, task YAML, reference artifact, and grader strategy.
• Adding hidden verifier patches/tests to starter tasks where behavior can be checked after the agent phase without revealing the checks in the prompt.
• Simplifying public graders so they do not duplicate hidden behavioral checks with brittle source-token assertions.
• Replacing raw string-token structural checks with semantic checks where the structure is truly required.
• Regenerating affected task cards.
• Running focused reference validation and at least one single-trial/single-job smoke run for a migrated representative task before broader validation.

Out of scope:

• Changing the hidden verifier infrastructure itself unless a clear bug blocks task migration.
• Adding an LLM grader.
• Building a general-purpose structural-check framework beyond the narrow helpers needed for these starter tasks.
• Solving ignored/generated artifact capture policy.
• Re-curating the starter suite’s task prompts or target repositories beyond what is necessary to keep migrated graders fair.

Dependencies

Hidden verifier patch support from issue #91 has landed.

Acceptance Criteria

• All current tasks/starter/* tasks are audited for behavioral grader brittleness and hidden-verifier suitability.
• Every audited starter task has one of:
  • a hidden verifier migration,
  • a semantic structural-check improvement,
  • or a short recorded rationale in the PR for why its current grader should remain as-is.
• Behavioral expectations that can be tested through hidden verifier patches no longer duplicate those checks with brittle source-token assertions.
• Remaining structural assertions are expressed semantically enough to accept equivalent implementations where practical.
• Task reports clearly distinguish behavioral verifier failures, structural failures, and artifact/scope hygiene failures.
• A representative migrated task is tested with one trial and one job (--trials 1 --jobs 1) before scaling validation.
• Task bundle validation and task-card drift checks pass.

Validation Plan

• Run focused unit/reference checks for changed task bundles and grader helpers.
• Run python3 -m agentlab task validate --check-task-cards tasks.
• Run git diff --check.
• Run one representative migrated task through python3 -m agentlab run ... --trials 1 --jobs 1.
• Broaden validation only after the single-trial path is known to be fair.

Deferred Follow-ups

• Any starter task that needs product/visual judgment or a nontrivial new structural-check framework should be split into a follow-up issue rather than quietly expanding this PR.
• Artifact/change-capture policy for ignored/generated files remains separate from this task-grader migration.

Design readiness: ready to implement
Consensus provenance: Codex discussion in this thread on 2026-06-26 after issue #91 landed; the human user agreed with the first-slice direction but explicitly corrected the scope to include a potential migration audit of all starter tasks, not only the three motivating examples.
Gate skipped: not applicable