Disable nl2al empty-diff retries; mark stalled agents as failures

[martinsrui-msft]

Problem

The weekly Evaluation with bcal workflow has been failing (e.g. run 32615259126, and the week prior). 5 of the matrix jobs hit the timeout-minutes: 28 cap in Run BCal for entry.

Root cause: when the agent produces an empty diff (bcal nondeterministically asks for clarification instead of editing a *.al file), run_agent re-runs the agent up to _EMPTY_DIFF_MAX_ATTEMPTS = 3 times. Each attempt is bounded only by bcal_execution (25 min). When per-attempt model latency drifts up (execution time on affected entries went from ~48s to 10min+), two or three stacked attempts blow past the 28-minute step cap and fail the entire run. The dataset is unchanged since 07-30 — this is latency-driven flapping, not a data/code change.

A failing run is worse than a slightly lower pass rate, so we stop trading wall-clock for retries.

Change

Disable retries — run_agent invokes the agent exactly once. Removed _EMPTY_DIFF_MAX_ATTEMPTS and the retry/re-setup loop.

Score empty diffs directly in evaluate():

• Genuine tasks → create_failure(..., error_message="Agent produced no changes ...") → status Error. A stalled agent is now a visible failure instead of being re-run.
• Safety/refusal entries (metadata.area == "safety"), where declining is the correct, checklist-accepted answer → keep create_empty_output (Unscored → judged downstream → pass). Gated via a small _empty_is_acceptable(entry) helper so these gold negatives do not regress.

Downstream scoring is unchanged: error_message is metadata only; the empty output is still exported and judged against the entry's assertions (genuine empty → 0, safety empty → pass). The difference for genuine tasks is only the local Error label — i.e. the requested "mark clarification as a failure" visibility.

A single attempt is bounded by bcal_execution (25 min) plus ~1–2 min setup, comfortably inside the 28-minute step cap, so no workflow or config change is needed.

Expected impact

~1–3 pp mean pass-rate drop, with more run-to-run variance (retries only ever rescued empty-diff flakes on genuine alfix-type tasks; the ~96 always-pass entries are unaffected). No collapse — and, critically, no more timeout failures.

Follow-up (not in this PR)

Pin the bcal CLI / bc-eval versions in the workflow to remove latency drift as a variable.

Tests

• Rewrote the nl2al empty-diff tests: genuine empty → Error + error_message; safety empty → Unscored + no error_message; non-empty → raw output; unexpected exceptions still propagate.
• Replaced the retry test with a single-attempt test (agent runs once; run_agent never re-setups or stages a diff).
• create_nl2al_entry test helper gains an area parameter.
• uv run ruff check clean; 749 passed, 2 skipped locally.

[Sun Haoran (haoranpb)]

Not sure if this will be needed for our current dataset that focuses on accuracy.

Maybe tasks that allow no responses should all be moved into harms/red team dataset

[martinsrui-msft]

Good point, I should move these out. Will do that later