Unifies local and API execution scripts.

Quick start

Set your API key:

export OPENAI_API_KEY=...           # or put this in a .env file
# optional (if you use a proxy/base override)
export OPENAI_BASE_URL=...

Install dependencies (either):

# Conda
conda env create -f environment.yml && conda activate inference-central

# or pip
pip install -r <(printf '%s\n' \
  openai \
  python-dotenv \
  tqdm \
  langchain \
  langgraph \
  langchain-openai \
  pydantic)

CLI

The unified CLI lives at api-infer/cli.py.

• Subcommands:

  • dir: run all *_prompts.json under a directory
  • file: run a single *_prompts.json file
  • subset: run a subset from one *_prompts.json via identifiers/indices/slice
  • baseline: convenience alias for dir --prompt-dir inputs/baseline

• Common flags:

  • --engine: batch (default), direct, or completions
  • --agentic: enable multi-round tool-using loop (works with both engines)
  • --max-rounds: max agentic rounds (default 8)
  • --tools and --tool-choice (auto|none|required)
  • --model, --completion-window (batch only), --max-output-tokens, --effort (low|medium|high)
  • --outputs-root, --run-tag, --quiet
  • --provider: auto (default), openai, or moonshot (override only; auto infers from --model)

• Multi-agent (manager as tools) flags (Agents SDK, direct or batch-agentic):

  • --subagent name=/abs/path/to/file_or_dir (repeatable): add a named sub‑agent. If a directory is provided, all *_prompts.json files inside are merged for that sub‑agent.
  • --subagent-dir /abs/path/to/dir (repeatable): load all *_prompts.json files in the directory; each file becomes a sub‑agent (name derived from the filename stem before _prompts).
  • --primary-agent-name: label used internally for the manager agent (default Main).

• Examples (from repo root):

  • Run all baseline prompts (Batch, single-shot):

    python -m api-infer.cli baseline

  • Run a directory with tools and agentic (Batch):

    python -m api-infer.cli dir --prompt-dir inputs/custom --tools --agentic --engine batch

  • Run one file (direct, no tools):

    python -m api-infer.cli file --prompt-file inputs/custom/behavior_action_sequencing_prompts.json --engine direct

  • Run first 5 prompts of a file with tools (agentic, Batch):

    python -m api-infer.cli subset \
      --prompt-file inputs/custom/behavior_action_sequencing_prompts.json \
      --indices 0 1 2 3 4 \
      --tools --agentic --engine batch

  • Slice by range (e.g., prompts 10..20 step 2):

    python -m api-infer.cli subset \
      --prompt-file inputs/custom/behavior_action_sequencing_prompts.json \
      --slice --start 10 --end 20 --step 2 \
      --engine batch

Moonshot (Kimi) support

• Provider is auto-inferred from --model. For Kimi models, the CLI switches to Moonshot's OpenAI-compatible endpoint. Use the completions engine.

  # Kimi K2 Thinking via Moonshot (auto provider inference)
  export MOONSHOT_API_KEY=sk-...   # or set OPENAI_API_KEY directly
  python -m api-infer.cli dir --prompt-dir inputs/custom --model kimi-k2-thinking --engine completions
  
  # Kimi K2 Thinking Turbo via Moonshot
  python -m api-infer.cli dir --prompt-dir inputs/custom --model kimi-k2-thinking-turbo --engine completions

• You can also set the base URL yourself:

  export OPENAI_API_KEY=sk-...
  export OPENAI_BASE_URL=https://api.moonshot.ai/v1
  python -m api-infer.cli dir --prompt-dir inputs/custom --model kimi-k2-thinking --engine completions
  python -m api-infer.cli dir --prompt-dir inputs/custom --model kimi-k2-thinking-turbo --engine completions

• Override provider only if needed:

  python -m api-infer.cli dir --prompt-dir inputs/custom --model kimi-k2-thinking --provider moonshot --engine completions
  python -m api-infer.cli dir --prompt-dir inputs/custom --model kimi-k2-thinking-turbo --provider moonshot --engine completions

Note: Agentic mode and local tools are not supported with Moonshot and are disabled automatically.

Reference: Moonshot Platform model list

Multi-agent: Manager (agents as tools)

• We support a central manager agent that handles all user interaction and invokes specialized sub‑agents exposed as tools. By default, all configured peer agents are exposed to the manager.

• Peer agent instruction files reuse the same schema as primary prompts: each JSON file is an array of { "identifier": string, "llm_prompt": string }. For a given prompt identifier, the peer agent’s llm_prompt is treated as that peer’s per‑item instructions.

• Reference: Manager (agents as tools) in the OpenAI Agents SDK docs: Agents as tools

• Examples:

  • File mode with two sub‑agents (manager calls sub‑agent tools as needed):

    python -m api-infer.cli file \
      --prompt-file inputs/custom/behavior_action_sequencing_prompts.json \
      --tools --agentic \
      --subagent Critic=/abs/path/to/critic_prompts.json \
      --subagent Planner=/abs/path/to/planner_prompts.json \
      --primary-agent-name Main

  • Dir mode with sub‑agents:

    python -m api-infer.cli dir \
      --prompt-dir inputs/custom \
      --tools --agentic \
      --subagent Critic=/abs/path/to/critic_prompts.json \
      --subagent Planner=/abs/path/to/planner_prompts.json \
      --primary-agent-name Main

  • Load multiple unnamed sub‑agents from a directory (each file becomes a tool):

    python -m api-infer.cli file \
      --prompt-file inputs/custom/behavior_action_sequencing_prompts.json \
      --tools --agentic \
      --subagent-dir /abs/path/to/subagents_dir \
      --primary-agent-name Main

Behavioral notes:

• If a sub‑agent lacks an entry for a particular identifier, it is excluded for that item.
• Tools defined in api-infer/tools.py are available to the manager and sub‑agents.
• The primary agent remains in control and calls sub‑agent tools; sub‑agents do not call each other.

Defaults

• model: gpt-5-mini
• completion-window (batch only): 24h
• max-output-tokens: 16384
• effort: medium (choices: low, medium, high)
• outputs-root: outputs/api-infer
• prompt-dir (dir/baseline): inputs/baseline
• engine: batch
• agentic: false
• max-rounds (when agentic): 8
• tools: false
• tool-choice: auto
• run-tag: auto UTC timestamp like YYYYMMDD-HHMMSS
• quiet: false

Environment:

• OPENAI_API_KEY: required (no default)
• OPENAI_BASE_URL: optional (SDK default if unset)

Outputs are written under outputs/api-infer/<run-tag>/{artifacts,submission} and paths are printed after each run.

Tool support (Batch API agentic loop)

What we implemented

• Multi-round agentic flow using only Batch: We preserve full conversation history and chain new Batch jobs between rounds when the model requests tools.
• Tools: Implemented as Agents SDK function tools in api-infer/tools.py via @function_tool and exposed with get_agents_tools().
• Runtime execution: The Agents SDK orchestrates tool calling internally. Tool call file logging has been removed.

Flow

1. Build a batch round with each item’s full messages history and tool specs (api-infer/oai_batch_run.py).
2. Submit Batch → poll → download output JSONL.
3. Parse assistant output and detect tool calls (extract_tool_calls).
4. Execute tools locally; append { "role": "tool", "tool_call_id": ..., "name": ..., "content": <result>} to history.
5. Repeat rounds until no tool calls are emitted (or max_rounds reached). Final submission is the last assistant message per item.

Key APIs:

• run_openai_batch_agentic(...) and run_openai_batch_agentic_from_dir(...) in api-infer/oai_batch_run.py.
• Tool definitions and executor in api-infer/tools.py.

Outputs

• Final predictions: outputs/api-infer/<run-tag>/submission/*_outputs.json (from final conversation state).
• Tool call audit log: outputs/api-infer/<run-tag>/artifacts/tool_calls.jsonl (aggregated across rounds).

Run a quick test (first 5 custom prompts with tools)

From repo root:

python -m api-infer.central test-custom-first5-tools

Or with the new CLI:

python -m api-infer.cli subset --prompt-file inputs/custom/behavior_action_sequencing_prompts.json --indices 0 1 2 3 4 --tools --agentic --engine batch

This uses the agentic Batch loop with tools enabled on inputs/custom/behavior_action_sequencing_prompts.json indices 0–4.

Add your own tools

• Create functions decorated with @tool in api-infer/tools.py (LangChain tools).
• They are auto-included by get_agents_tools() and passed directly to the Agents SDK Agent.

Notes

• Batch requests are single-shot; multi-turn is achieved by submitting subsequent batches with updated messages.
• Tool schema uses top-level name and parameters per Responses API.
• “Judge LM” pattern: implement a tool that calls your judge model and returns its verdict; it will be executed locally between rounds and its output fed into the next Batch round.