Terminal Bench Integration into rLLM (Simplified)

[JasonWei05]

PR Review: Terminal-Bench Terminus 1 Integration (rLLM)

Summary

This PR integrates Terminal Bench’s Terminus 1 agent into rLLM for reproducible terminal task evaluation. We reuse Terminal Bench internals (prompt templates, command execution, evaluation) and only replace the LLM interaction with any RolloutEngine (OpenAI/VERL). The workflow returns a standard rLLM Episode with a single "terminus" trajectory containing step-by-step chat history and model responses.

Requirements: Python ≥ 3.12, pip install terminal-bench, and OPENAI_API_KEY set. Run the example with:

python examples/terminal/run_terminal.py

Files

• rllm/rllm/integrations/terminal_terminus_1.py

  • Adds RLLMModel, a thin adapter subclassing Terminal-Bench Terminus.
  • Maintains TB Chat-style message history, formats JSON response schema, retries at most three times, parses into CommandBatchResponse.
  • Runs the agent loop and returns (Trajectory, TerminationReason) for packaging by the workflow.

• rllm/examples/terminal/terminus_workflow.py

  • Adds TerminalTerminusWorkflow using RLLMModel and Terminal-Bench’s terminal/session management.
  • Resets environment, runs the control loop to completion, evaluates by running Terminal-Bench tests, returns an Episode.

• rllm/examples/terminus/run_terminus.py (example runner)

  • Demonstrates running the full Terminal-Bench dataset with AgentWorkflowEngine using the RolloutEngine.
  • Keeps the engine interchangeable.

• rllm/examples/terminus/prepare_terminal_data.py (dataset helper)

  • Loads Terminal-Bench datasets via terminal_bench.dataset.Dataset and extracts {task_path, task_id, instruction}.

[jeffreysijuntan]

Great work!