A provider error mid-run is graded as a real 0, and the job reports no errors #717

What happens. If the model provider fails during the agent loop (429, 5xx, 529 Overloaded, a dropped stream after retries), ToolAgent._loop catches the exception and only sets trace.status = "error" (hud/agents/tool_agent.py:263-266). The run is then graded anyway with {"answer": None} (hud/eval/run.py:343-350), and Job.reward counts any run that has a grade (hud/eval/job.py:65-69):

graded = [run.reward for run in self.runs
          if not run.grade.is_error and (run.grade.raw or not run.trace.is_error)]

Job.errors uses the mirror condition, so these runs never show up as errors.

Repro (offline, fake transport). One task where every model call returns 529, one task that scores 1.0:

grader called with: {'answer': None}
trace.status: error | stop_reason: None
grade: 0.0 is_error: False
Job.reward: 0.5 | Job.errors: []

With all runs hitting 529, hud eval prints Mean reward: 0.000 and no "Errors" line.

Why it matters. The docstring on Job.reward and the test at hud/eval/tests/test_job.py:112 say infrastructure failure must not drag the mean down as a silent zero, but that test only covers Run.failed (failure before launch). PR #504 handled the timeout case (stop_reason="timeout"). A partial provider outage currently lowers a model's score without any visible signal. Whether a run counts also depends on how the template's grader handles answer=None: a grader that guards if answer and ... turns the outage into 0.0, one that calls answer.strip() raises and the run is excluded.

Possible fix. In the except branch of _loop, classify provider/transport errors (anthropic.APIError, openai.APIError, google.genai.errors.APIError, httpx transport errors) into a distinct stop reason such as "provider_error", skip grading or mark the grade as an error for those runs, and list them in Job.errors. Happy to send a PR once you pick the policy (exclude and count as error, or retry).

Found while reading the SDK at 67e0ffa (hud 0.6.20); the repro runs offline with a fake Anthropic transport and a stub grader, so no API key or hud.ai access is involved.