A lightweight, practical evaluation framework for AI agents in production.
Not a benchmark suite. Not an academic leaderboard. A framework for answering: "Is my agent getting better or worse at its actual job?"
Most AI evaluation tools measure model capabilities (MMLU, HumanEval, etc.). That's useful but doesn't answer the question you actually care about in production: is your agent reliably completing the tasks you need it to do?
This framework gives you:
- A scoring rubric for manual evaluation (15 min/week)
- Golden set templates for common agent types
- Automated metrics collection from session logs
- A weekly dashboard that tracks trends over time
# Copy the framework into your project
cp -r templates/ your-project/evals/
# Edit golden sets for your agents
vim your-project/evals/golden-sets/your-agent.md
# Run your first metrics collection
bash your-project/evals/scripts/collect-metrics.sh
# Fill in manual scores (15 min)
vim your-project/evals/results/$(date +%Y)-W$(date +%V).mdtemplates/
├── METHODOLOGY.md # Scoring rubric and review process
├── golden-sets/
│ ├── engineer.md # For code-writing agents
│ ├── ops.md # For infrastructure/operations agents
│ ├── research.md # For research/analysis agents
│ └── orchestrator.md # For meta-orchestrator/router agents
├── results/
│ └── YYYY-WXX.md.template # Weekly results template
├── scripts/
│ └── collect-metrics.sh # Automated metrics from logs
└── dashboards/
└── weekly-report.md.template # Dashboard template
Three dimensions, each scored 1-5:
| Score | Meaning |
|---|---|
| 5 | Perfect. Output ready to use, no corrections. |
| 4 | Minor issues. Small tweaks needed. |
| 3 | Partial success. Core done but significant gaps. |
| 2 | Mostly failed. Some useful output but needs major rework. |
| 1 | Complete failure. Wrong approach or refused valid task. |
| Score | Meaning |
|---|---|
| 5 | Fully autonomous. Figured out context, no questions needed. |
| 4 | One reasonable clarifying question. |
| 3 | Multiple rounds of clarification. |
| 2 | Needed step-by-step guidance. |
| 1 | Could not proceed without constant intervention. |
| Score | Meaning |
|---|---|
| 5 | Minimal tokens/time for the complexity. |
| 4 | Slight overhead but reasonable. |
| 3 | Notable waste. Wrong paths explored. |
| 2 | Significant waste. Long detours, repeated attempts. |
| 1 | Extreme waste. Loops or irrelevant work. |
A golden set is a curated collection of representative tasks for an agent type. Each includes:
- 5 routine tasks (common, should score 5 reliably)
- 5 challenging tasks (complex, tests limits)
- 3 edge cases (ambiguous inputs, conflicting requirements)
- 2 regression checks (tasks that previously failed)
See templates/golden-sets/ for ready-to-customize examples.
A task is one coherent request, even if it spans multiple messages.
Minimum threshold: would take a human >5 minutes to do manually. Skip trivial queries.
The collect-metrics.sh script gathers:
- Session counts per agent
- Task entry counts from memory/log files
- Incident keyword frequency (error, fix, broken, crash, etc.)
Adapt the script paths for your setup. It's intentionally simple (bash + grep) so you can run it anywhere.
Every Friday:
- Review agent sessions (start with highest-volume agent)
- Score 3-5 representative tasks using the rubric
- Log scores in
results/YYYY-WXX.md - Run
collect-metrics.shfor automated numbers - Update the dashboard
Monthly (first Monday, 30 minutes):
- Compare weekly scores for trends
- Identify regressions
- Note which task types score lowest
- Refresh golden sets if task mix has shifted
This framework is designed to be adapted. The templates are starting points:
- Change the agent types to match your system
- Adjust golden sets with your real tasks
- Modify the metrics script for your log format
- Add dimensions if you need them (security, creativity, etc.)
The only hard requirement: be consistent. Use the same rubric every week so scores are comparable over time.
- Lightweight over comprehensive. 15 minutes/week beats a sophisticated system nobody uses.
- Manual scoring is valuable. Automated evals miss judgment quality. Your assessment of "was this actually useful?" matters.
- Trends over absolutes. A score of 3.5 means nothing in isolation. A drop from 4.2 to 3.5 after a prompt change tells you something.
- Golden sets prevent cherry-picking. Evaluating the same task types consistently reveals patterns.
Built from running 9 AI agents in production since February 2026. Methodology refined through weekly use. Every template reflects real evaluation needs, not theoretical frameworks.
MIT