Skip to content

Preserve artifact criteria on agent ERROR and warn on ineffective task_timeout #114

Description

@tmatup

Problem

With coder-eval 0.9.5, an agent iteration that ends in ERROR skips every success criterion. The result is an empty criterion result set and a 0.0 score even when deterministic artifact-only criteria can be evaluated against the preserved workspace.

In the measured configuration:

  • iterations: 1
  • max_turns: 120
  • turn_timeout: 1200
  • task_timeout: 1500

Two preserved SDK workspaces ended at the single-iteration boundary. At that point, both artifacts passed all four scored artifact greps offline and also passed uip maestro flow validate. Because the terminal state was ERROR, coder-eval evaluated none of those criteria and recorded 0.0.

This conflates two different facts:

  1. the agent did not terminate cleanly; and
  2. the artifact was absent or failed its deterministic checks.

There is a coupled configuration issue: for a single-iteration agent, turn_timeout is the de-facto agent budget. A larger task_timeout cannot bind first and is therefore dead configuration for the agent iteration.

Requested behavior

  1. Keep ERROR as the terminal iteration status, but evaluate deterministic artifact-only criteria when the preserved workspace is readable, or represent them separately from criteria that truly cannot run after ERROR.
  2. Distinguish “criterion failed” from “criterion was not evaluated.”
  3. Warn or reject when a single-iteration agent config sets task_timeout above the only iteration's turn_timeout, because the larger value cannot govern that iteration.

Scope

This is a measurement-fidelity request for future evaluation blocks. The behavior affected both arms symmetrically and does not change the campaign conclusion.

This is not a request to rescore any historical run.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions