custom-agents.md

Custom Agents

NatureBench is harness-agnostic at the protocol level. Every task runs in a container with read-only data and a host-side evaluation service; the agent writes outputs and submits them over HTTP for a SOTA-normalized score. The built-in Claude Code, Codex, and Gemini CLIs are wired in through the same adapter interface that a custom agent can use.

There are two ways to plug in a custom agent. Pick based on whether you want NatureBench to orchestrate the run for you.

Path	You provide	NatureBench provides
A. Protocol-only	Your agent + your own runner	The task packages and the evaluation protocol
B. Adapter	An AgentAdapter subclass	Container orchestration, evaluation service, post-hoc judge

---

The evaluation protocol

This section documents how to use the evaluation service that NatureBench provides; a custom agent must adapt to this interface to obtain a comparable score.

Inside the task container the agent is given:

• DATA_DIR=/task/problem/data — read-only inputs (one sub-folder per instance).
• OUTPUT_DIR=/workspace/output — where outputs are written, one sub-folder per instance.
• EVAL_SERVICE_URL — base URL of the host-side eval service.

NatureBench provides each task run with a random opaque token instead of exposing task identifiers that could reveal the source paper. The agent must include this token in each evaluation service request so that the service can identify the corresponding task and batch. The token is supplied when the task run is registered with the evaluation service, before the agent starts.

The evaluation service provides four commands for the agent:

• Evaluate the output and record a new scoring attempt:

  curl -s -X POST "$EVAL_SERVICE_URL/evaluate" \
    -H "Content-Type: application/json" \
    -d '{"eval_token": "<opaque-token>"}'

• Return the best score across all completed attempts:

  curl -s "$EVAL_SERVICE_URL/best_score?eval_token=<opaque-token>"

• Return the elapsed, remaining, and total allowed solve time:

  curl -s "$EVAL_SERVICE_URL/time_remaining?eval_token=<opaque-token>"

• Check whether the evaluation service is available:

  curl -s "$EVAL_SERVICE_URL/health"

The built-in CLIs receive these evaluation-service instructions in their prompts. A custom agent must receive the same instructions.

---

Path A — protocol-only

1. Download the task packages with the official script:

   python run_naturebench.py --dataset-id FrontisAI/NatureBench --tasks all --download-only

2. Run your own runner against the downloaded task packages. The orchestration is up to you, but to keep scores comparable with the built-in agents we recommend reusing — or at least referencing — NatureBench's own evaluation service and evaluation environment:

   • Evaluation service — reuse eval_service.py (started via scripts/start_eval_services.sh, see docs/configuration.md) so scoring, timing, and best-score tracking match the official protocol. Your agent then submits to /evaluate exactly as described in The evaluation protocol.
   • Evaluation environment — run your agent inside the NatureBench Docker setup (naturebench-base:v3 plus each task's environment/Dockerfile.v3) rather than an arbitrary local environment, so running environment matches the reported runs.

---

Path B — write an adapter

An adapter tells solve.py how to launch your agent inside the standard task container and (optionally) where its solve/iteration history lives for the post-hoc judge. The interface lives in agent/adapter.py; the built-in CLIs in agent/cli_adapters.py are worked examples.

A minimal adapter only implements system_prompt and build_command:

# agent/myagent_adapter.py
from typing import List
from agent.adapter import REGISTRY, AgentAdapter, AgentRunContext
from agent.claude import ClaudeAgent  # reuse the built-in task prompt, optional


class MyAgentAdapter(AgentAdapter):
    name = "myagent"            # the value passed to --agent

    def system_prompt(self, ctx: AgentRunContext) -> str:
        # Reuse the standard task prompt (includes the eval protocol), or build
        # your own (it must write run.py / output and submit to $EVAL_SERVICE_URL/evaluate).
        # ctx carries the opaque eval token and the service URL.
        return ClaudeAgent(model_name=ctx.model, mode=ctx.mode).build_system_prompt({
            "eval_service_url": ctx.eval_service_url,
            "eval_token": ctx.eval_token,
            "time_limit_minutes": ctx.time_limit_minutes,
        })

    def build_command(self, ctx: AgentRunContext) -> List[str]:
        # The argv run inside the container; any executable form works
        # (a binary, `python -m ...`, a shell script, ...).
        # If your agent program is not already in the image, you can install it at
        # runtime here, or bake it into the image (see "Container image" below).
        return ["my-agent", "--model", ctx.model, "--prompt", ctx.system_prompt]


REGISTRY.register(MyAgentAdapter())

Optional hooks (all default to a no-op, so a minimal agent needs none):

• docker_mounts(ctx) / extra_env(ctx) — add extra -v src:dst mounts or -e KEY=VALUE variables to docker run. Use these when your agent needs a host path inside the container or an environment variable beyond the Anthropic/OpenAI/Gemini keys solve.py already forwards. /task/problem (read-only) and /workspace are mounted for every agent regardless.
• transcript_path(task_out_dir) / excerpt_transcript(...) — expose your agent's solve/iteration history to the post-hoc judge (see "Judge history" below).

Selecting your agent

The registry is populated by importing the adapter module. Import it once before solve.py dispatches (e.g. add an import line to agent/__init__.py), then select it:

python run_naturebench.py --agent myagent --model <model> --tasks cpu

An unknown --agent is rejected with the list of registered agents.

Container image

The built-in CLIs are baked into the NatureBench base image. If your agent program (binary, Python package, or script) is not already in the image, make it available one of two ways:

• Runtime install — install it inside build_command (e.g. ["bash", "-lc", "pip install my-agent && my-agent ..."]).
• Extend the image — build a derived image from the NatureBench base image (docker/Dockerfile.base) with your agent installed. How you run it then depends on its tag:
  • if you tag it as the base name the task Dockerfiles build FROM, both run paths work — the default --skip-build, and per-task image builds via --build-task-images;
  • otherwise, run it with --skip-build --base-image <your-image>.

Judge history

After a run, the post-hoc validity judge reviews the agent's workspace code and, when available, its solve/iteration history. Two hooks control this:

• transcript_path(task_out_dir) tells the judge where your agent's solve/iteration history log is. The built-in adapters return their internal session file or streamed stdout log. Returning None (the default) means the judge reviews only the final workspace code snapshot.
• excerpt_transcript(log_path, *, max_bytes, focus_start, focus_end) provides a parser for a custom history-log format. The runner supplies the same byte budget and SCORE_ATTEMPT focus window available to the built-in parser. A custom implementation may use any subset of these inputs, but it must accept all runner-supplied keyword arguments, either explicitly or through **kwargs. Returning None (the default) falls back to the built-in parser, whose format sniffing recognizes the Claude/Codex/Gemini CLI logs. After either parser returns, the framework clips its output to MAX_LOG_EXCERPT_BYTES (2,000,000 UTF-8 bytes) before adding it to the judge prompt. Parsers may use max_bytes for format-aware selection, but the framework enforces the final hard limit.