Fix the eval suite so it measures something, and add the missing runner

[ericjohanson-langchain]

Two of the three evaluators were decoupled from agent behaviour. Nothing caught it because no runner existed — the suite had never been executed.

Fed a simulated perfect agent, outputs byte-identical to ground truth:

severity_accuracy mean: 0.000   (should be 1.0)
tool_coverage     mean: 1.000

Fed a deliberately wrong agent, tool_coverage still returned 1.000.

What was broken

Evaluator	Defect
severity_accuracy	matched only [CRITICAL]; the dataset writes CRITICAL:. Neither side extracted, returned 0 every time
tool_coverage	read expected_trajectory, a key the dataset never defines. Short-circuited to 1.0 unconditionally
response_quality	built ChatOpenAI(gpt-4o-mini) with langchain-openai absent from requirements and OPENAI_API_KEY configured nowhere; also annotated a Pydantic model then subscripted it as a dict

severity_accuracy now reads the typed overall_severity when a HealthReport is present and accepts bracketed, colon-suffixed, and bare leading forms otherwise, covering both code paths. A missing reference is now unscored rather than zero, so a gap in the dataset stops looking like an agent failure. The judge moved to Anthropic Haiku — this repo already has that credential, so no second provider and no second key.

The output contract is now documented at the top of evaluators.py, since reading the agent output under the ground-truth field name expected_response is exactly what caused two of these defects.

Online evaluators were a separate problem

They ran as hand-copied reference-based functions against production traces, which carry no ground-truth example. So they compared "" against "" and recorded a constant severity_accuracy: 0 and tool_coverage: 1.0 on every live run. Reference-based metrics are structurally unable to work online.

Now only reference-free evaluators upload, and their source comes from evaluators.py via inspect.getsource rather than a second hand-maintained copy. Verified the generated source executes in an empty namespace with no repo imports, which is all the sandbox provides.

Two reference-free evaluators, both grounded in failures this agent actually shipped:

• deferred_work — catches a report handing the question back to the operator ("check pod CPU/memory metrics"), which is what the missing utilization data produced for weeks
• finding_specificity — whether findings name a concrete object, which also gates whether they can be fingerprinted at all

Known limitation, recorded in the runner

The dataset describes fictional resources, so the agent read tools return not-found. tool_coverage measures tool selection honestly; response_quality is grading diagnosis built on failed lookups and will read lower than the agent deserves. Treat it as a floor until the tools are stubbed to return each scenario's implied state. That is the next obvious improvement to this dataset.

Tests

25 on the evaluators themselves, including one asserting no reference-based evaluator returns the same score for a good and a bad agent. That ten-line check would have caught every defect above the day it was written. 189 passing overall.

Still blocked

run_eval.py and the uploader both need LANGSMITH_API_KEY. LANGSMITH_RUNS_ENDPOINTS carries keys for trace export only; the dataset client does not consult it. Both scripts now fail with that explanation rather than a generic auth error.