Add RolloutGateway to enable native CLI agent training

[rasdani]

Adds a rollout gateway to the vLLM inference server, enabling server-side multi-turn rollout execution.

depends on PrimeIntellect-ai/verifiers#954

Rollout gateway (rollout_gateway.py, 542 lines — new file):

• Register/unregister rollouts with per-rollout state, locks, and trajectory tracking
• Proxy chat/completions calls through the gateway with DP-rank-aware routing
• Token-level trajectory extraction (prompt/completion ids, masks, logprobs)
• Pydantic request/response models for type-safe API

Inference server (server.py):

• Mount rollout gateway router alongside existing chat endpoints
• Initialize RolloutRegistry in custom_init_app_state (disabled when api_server_count > 1)

Config (config.py, inference/config.py):

• log_rollout_gateway_turns on inference config

---

Note

Medium Risk
Adds new FastAPI endpoints and per-rollout state management inside the vLLM inference server, plus tweaks DP/api-server scaling behavior; issues here could break inference serving or rollout collection in production.

Overview
Introduces a new inference-side rollout gateway (/v1/rollouts/...) that lets clients register multi-turn rollouts, route requests to a data-parallel rank, and fetch token-level trajectories (ids/masks/logprobs), with optional verbose turn logging.

Updates inference config to add log_rollout_gateway_turns and auto_scale_api_servers (gating the existing dp→api_server_count auto-scaling), and mounts/initializes the gateway in the vLLM server (disabled when api_server_count>1). Also updates orchestrator metric logging to pass an explicit step.

^{Written by Cursor Bugbot for commit 27806a4. This will update automatically on new commits. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.}