Problem
With coder-eval 0.9.5, an agent iteration that ends in ERROR skips every success criterion. The result is an empty criterion result set and a 0.0 score even when deterministic artifact-only criteria can be evaluated against the preserved workspace.
In the measured configuration:
iterations: 1
max_turns: 120
turn_timeout: 1200
task_timeout: 1500
Two preserved SDK workspaces ended at the single-iteration boundary. At that point, both artifacts passed all four scored artifact greps offline and also passed uip maestro flow validate. Because the terminal state was ERROR, coder-eval evaluated none of those criteria and recorded 0.0.
This conflates two different facts:
- the agent did not terminate cleanly; and
- the artifact was absent or failed its deterministic checks.
There is a coupled configuration issue: for a single-iteration agent, turn_timeout is the de-facto agent budget. A larger task_timeout cannot bind first and is therefore dead configuration for the agent iteration.
Requested behavior
- Keep
ERROR as the terminal iteration status, but evaluate deterministic artifact-only criteria when the preserved workspace is readable, or represent them separately from criteria that truly cannot run after ERROR.
- Distinguish “criterion failed” from “criterion was not evaluated.”
- Warn or reject when a single-iteration agent config sets
task_timeout above the only iteration's turn_timeout, because the larger value cannot govern that iteration.
Scope
This is a measurement-fidelity request for future evaluation blocks. The behavior affected both arms symmetrically and does not change the campaign conclusion.
This is not a request to rescore any historical run.
Problem
With coder-eval 0.9.5, an agent iteration that ends in
ERRORskips every success criterion. The result is an empty criterion result set and a 0.0 score even when deterministic artifact-only criteria can be evaluated against the preserved workspace.In the measured configuration:
iterations: 1max_turns: 120turn_timeout: 1200task_timeout: 1500Two preserved SDK workspaces ended at the single-iteration boundary. At that point, both artifacts passed all four scored artifact greps offline and also passed
uip maestro flow validate. Because the terminal state wasERROR, coder-eval evaluated none of those criteria and recorded 0.0.This conflates two different facts:
There is a coupled configuration issue: for a single-iteration agent,
turn_timeoutis the de-facto agent budget. A largertask_timeoutcannot bind first and is therefore dead configuration for the agent iteration.Requested behavior
ERRORas the terminal iteration status, but evaluate deterministic artifact-only criteria when the preserved workspace is readable, or represent them separately from criteria that truly cannot run afterERROR.task_timeoutabove the only iteration'sturn_timeout, because the larger value cannot govern that iteration.Scope
This is a measurement-fidelity request for future evaluation blocks. The behavior affected both arms symmetrically and does not change the campaign conclusion.
This is not a request to rescore any historical run.