Skip to content
BranchesTags

Repository files navigation

AgentBench

AI Agent Evaluation Platform with Multi-Agent Orchestration and Performance Benchmarking

AgentBench is a comprehensive framework for evaluating, benchmarking, and monitoring AI agents. Built with LangGraph orchestration, it provides production-ready tools for assessing agent performance across multiple dimensions including accuracy, cost, latency, and quality.

Features

  • Multi-Agent Orchestration - LangGraph-powered workflow management (serial, parallel, conditional)
  • Comprehensive Evaluation - 6+ evaluation metrics with customizable weights
  • Transcript Verification - Regex-based pattern matching for output validation
  • Real-time Monitoring - WebSocket server for live agent tracking
  • Performance Benchmarking - Comparative analysis and ranking system
  • MCP Protocol Integration - Standardized agent communication
  • Metrics Storage - ChromaDB for transcripts, Redis for caching
  • Interactive Dashboard - Streamlit-based visualization
  • Docker Support - Containerized deployment with compose

Architecture

agentbench/
├── orchestration/          # Multi-agent coordination
│   ├── conductor.py        # Main orchestrator
│   ├── agents/            # Agent implementations
│   └── workflows/         # Workflow types
├── evaluation/            # Evaluation framework
│   ├── framework.py       # Core evaluation engine
│   ├── metrics/          # Individual metrics
│   ├── verifier.py       # Transcript verification
│   ├── scoring.py        # Score calculation
│   └── benchmarking.py   # Comparative benchmarking
├── mcp/                  # MCP Protocol implementation
│   ├── server.py         # MCP server
│   └── tools/           # File ops, web search, database
├── monitoring/           # Real-time monitoring
│   ├── websocket_server.py
│   ├── metrics_collector.py
│   └── dashboard.py      # Streamlit dashboard
└── main.py              # FastAPI application

Quick Start

Prerequisites

  • Python 3.11+
  • Docker and Docker Compose
  • Redis (for caching)
  • PostgreSQL (for results storage)

Installation

  1. Clone the repository:
git clone <repository-url>
cd agentbench
  1. Create virtual environment:
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
  1. Install dependencies:
pip install -r requirements.txt
  1. Set environment variables:
export OPENAI_API_KEY=your_key_here
export ANTHROPIC_API_KEY=your_key_here
export REDIS_HOST=localhost
export DATABASE_URL=postgresql://user:pass@localhost:5432/agentbench

Running with Docker

# Start all services
docker-compose up -d

# View logs
docker-compose logs -f

# Stop services
docker-compose down

This will start:

  • FastAPI backend on port 8000
  • WebSocket server on port 8765
  • Streamlit dashboard on port 8501
  • Redis on port 6379
  • PostgreSQL on port 5432

Running Locally

# Start the API server
python main.py

# In another terminal, start the WebSocket server
python -m monitoring.websocket_server

# In another terminal, start the dashboard
streamlit run monitoring/dashboard.py

Usage

1. Register Agents

import requests

response = requests.post("http://localhost:8000/agents/register", json={
    "name": "my_research_agent",
    "type": "research",
    "model": "gpt-4",
    "temperature": 0.7
})

print(response.json())

2. Create Workflow

response = requests.post("http://localhost:8000/workflows/create", json={
    "workflow_name": "research_workflow",
    "agent_sequence": ["my_research_agent"],
    "workflow_type": "serial"
})

3. Execute Task

response = requests.post("http://localhost:8000/tasks/execute", json={
    "workflow_name": "research_workflow",
    "task_description": "Research current trends in AI agents",
    "task_type": "research"
})

task_id = response.json()["task_id"]

4. Run Evaluation

response = requests.post("http://localhost:8000/evaluations/run", json={
    "task_id": task_id,
    "agent_id": "agent_id_here",
    "criteria": {
        "max_latency_ms": 10000,
        "max_cost_usd": 0.50
    }
})

print(response.json())

5. Compare Agents

response = requests.post("http://localhost:8000/evaluations/compare",
    json=["agent_id_1", "agent_id_2"]
)

comparison = response.json()
print(f"Best overall: {comparison['best_overall']}")

Evaluation Metrics

AgentBench evaluates agents across 6 key dimensions:

  1. Accuracy (25%) - Correctness against expected outputs
  2. Quality (20%) - Output clarity, completeness, and structure
  3. Latency (15%) - Response time and execution speed
  4. Cost Efficiency (15%) - Token usage and API costs
  5. Consistency (15%) - Performance stability across runs
  6. Error Handling (10%) - Recovery and resilience

Each metric is scored 0-10, with an overall weighted score and letter grade (A+ to F).

Agent Types

Research Agent

Specialized in information gathering and analysis.

from orchestration.agents.research_agent import ResearchAgent
from orchestration.agents.base_agent import AgentConfig

config = AgentConfig(name="researcher", model="gpt-4")
agent = ResearchAgent(config)
await agent.initialize()

Coding Agent

Generates and modifies code across multiple languages.

from orchestration.agents.coding_agent import CodingAgent

config = AgentConfig(name="coder", model="gpt-4", temperature=0.5)
agent = CodingAgent(config)

Review Agent

Assesses quality and provides feedback.

from orchestration.agents.review_agent import ReviewAgent

config = AgentConfig(name="reviewer", model="gpt-4", temperature=0.3)
agent = ReviewAgent(config)

Executor Agent

Runs commands and validates outputs.

from orchestration.agents.executor_agent import ExecutorAgent

config = AgentConfig(name="executor")
agent = ExecutorAgent(config)

Workflow Types

Serial Workflow

Agents execute sequentially.

await conductor.create_workflow(
    workflow_name="serial_example",
    agent_sequence=["agent1", "agent2", "agent3"],
    workflow_type="serial"
)

Parallel Workflow

Agents execute simultaneously.

await conductor.create_workflow(
    workflow_name="parallel_example",
    agent_sequence=["agent1", "agent2", "agent3"],
    workflow_type="parallel"
)

Conditional Workflow

Dynamic routing based on conditions.

await conductor.create_workflow(
    workflow_name="conditional_example",
    agent_sequence=["agent1", "agent2", "agent3"],
    workflow_type="conditional"
)

Transcript Verification

Define validation rules to verify agent outputs:

from evaluation.verifier import TranscriptVerifier, VerificationRule

verifier = TranscriptVerifier()

verifier.add_rule(
    task_type="research",
    rule=VerificationRule(
        name="contains_findings",
        pattern=r"found|discovered|identified",
        required=True,
        description="Research produced findings"
    )
)

results = await verifier.verify_transcript("research", transcript)

Benchmarking

Run comparative benchmarks across multiple agents:

from evaluation.benchmarking import ComparativeBenchmark

benchmark = ComparativeBenchmark()

# Create benchmark suite
suite = benchmark.create_suite(
    suite_id="suite_001",
    name="Research Tasks",
    description="Standard research task suite",
    tasks=[...]
)

# Run benchmark
results = await benchmark.run_benchmark(
    agent_ids=["agent1", "agent2"],
    suite_id="suite_001"
)

# Get rankings
rankings = benchmark.rank_agents(by="overall")

Dashboard

Access the Streamlit dashboard at http://localhost:8501 to:

  • View agent performance metrics
  • Compare multiple agents
  • Monitor real-time activity
  • Analyze performance trends
  • Export reports

API Endpoints

Endpoint Method Description
/agents/register POST Register new agent
/agents GET List all agents
/workflows/create POST Create workflow
/tasks/execute POST Execute task
/tasks/{task_id} GET Get task status
/evaluations/run POST Run evaluation
/evaluations/agent/{agent_id} GET Get agent evaluations
/evaluations/compare POST Compare agents
/metrics/agent/{agent_id} GET Get agent metrics

MCP Protocol

The MCP server provides standardized tools:

from mcp.server import mcp

# Execute agent task
result = await mcp.execute_agent_task(
    agent_name="research_agent",
    task_description="Research AI trends",
    task_type="research"
)

# List agents
agents = await mcp.list_available_agents()

# Get evaluation results
results = await mcp.get_evaluation_results(task_id="task_123")

Testing

Run test scenarios:

# Run full evaluation suite
python environments/tasks/test_scenarios.py

# Run specific tests
pytest tests/ -v

Configuration

Customize evaluation weights in task_definitions.yaml:

evaluation_metrics:
  weights:
    accuracy: 0.25
    quality: 0.20
    latency: 0.15
    cost_efficiency: 0.15
    consistency: 0.15
    error_handling: 0.10

Tech Stack

  • Orchestration: LangGraph, LangChain
  • Backend: FastAPI, Uvicorn
  • Database: PostgreSQL, ChromaDB, Redis
  • Monitoring: WebSockets, Streamlit
  • Protocol: FastMCP
  • LLMs: OpenAI, Anthropic
  • Container: Docker, Docker Compose

Project Structure

agentbench/
├── orchestration/           # Agent orchestration
├── evaluation/             # Evaluation framework
├── mcp/                   # MCP protocol
├── monitoring/            # Monitoring & dashboard
├── environments/          # Docker & tasks
├── tests/                # Test suite
├── main.py              # FastAPI app
├── requirements.txt     # Dependencies
├── docker-compose.yml   # Docker config
└── README.md           # Documentation

Contributing

  1. Fork the repository
  2. Create feature branch (git checkout -b feature/amazing-feature)
  3. Commit changes (git commit -m 'Add amazing feature')
  4. Push to branch (git push origin feature/amazing-feature)
  5. Open Pull Request

License

MIT License - see LICENSE file for details.

Roadmap

  • Support for more LLM providers (Cohere, Mistral)
  • Advanced workflow patterns (loops, retries)
  • Multi-modal agent support
  • Distributed agent execution
  • GraphQL API
  • Mobile dashboard
  • A/B testing framework
  • Cost optimization suggestions

Support

For issues and questions:

  • GitHub Issues: [Create issue]
  • Documentation: [Wiki]
  • Discord: [Join community]

Acknowledgments

Built with:

  • LangGraph by LangChain
  • FastMCP by Anthropic
  • FastAPI by Sebastián Ramírez
  • Streamlit

AgentBench - Professional AI Agent Evaluation Platform

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages