Python benchmark framework comparing agent orchestration architectures on synthetic tasks only.
Scope V1 (frozen):
- synthetic benchmark only
- no real GPU integration
- no web UI
- no distributed runtime
- no real vLLM dependency
- proxy-based inference cost/latency/context modeling only
- focus on orchestration, evals, and ablations
Entry points (after you run pip install -e .):
- Run benchmark:
python scripts/run_benchmark.py --config configs/experiment_v1.json - Aggregate results:
python scripts/aggregate_results.py --traces-dir ./outputs/traces