evalstack

Open-source LLM evaluation for teams that don't want to pay $300/month for Braintrust.

Live demo: evalstack.kartikaneja.com Status: alpha · last shipped 2026-06-22 Built by: Kartik Aneja — AI/ML Platform Engineer

---

Why this exists

You ship an LLM feature. You iterate on the prompt. You ship again. Last week's regression resurfaces. You have no idea.

See PRODUCT.md for the full writeup. TL;DR:

• Who: AI Engineer at a 5–50 person startup running OpenAI or Anthropic in production
• Pain: No CI for prompts. No regression detection. Excel sheets and Slack screenshots.
• Why now: LLM eval is the #1 underserved 2026 topic. Braintrust raised $36M but is closed + paid.

Demo

Runs list — every eval run reported by the SDK or CLI, with a "Compare two runs" picker at the bottom:

Run-level diff — judge means with deltas, 10-bucket score histograms per judge (A in zinc, B in indigo), top regressions/improvements panels, and a per-prompt matched table with drill-down to event-level diff:

Run detail — every event in the run with per-event judge scores + reasoning + a compare-events picker at the bottom:

Event-level diff — word-level LCS diff on prompts + completions plus per-judge A-vs-B deltas:

What works today (alpha MVP)

• Python SDK — one decorator (@evalstack.trace) captures LLM calls + outputs
• FastAPI server — receives traces, runs configured judges, stores results in SQLite (Postgres next)
• LLM-as-judge — accuracy + helpfulness rubrics out of the box, fully extensible
• CLI — evalstack run <eval.yaml> runs a suite of eval cases and prints a diff
• Web dashboard — Next.js + Tailwind UI for browsing runs, events, judge scores, and side-by-side event diffs

Try it (60 seconds, local)

git clone https://github.com/anejakartik/evalstack.git
cd evalstack
pip install -e ./sdk
export OPENAI_API_KEY=sk-...
docker compose up -d              # starts FastAPI server (:8000) + Next.js dashboard (:3000)
python examples/quickstart.py     # runs a sample eval
open http://localhost:3000        # browse runs in the dashboard

Architecture

See docs/architecture.md. Stack: Python + FastAPI + SQLite (→ Postgres) + Next.js dashboard, deployed on Fly.io + Vercel.

What's next

See ROADMAP.md. Top items: web dashboard, Postgres backend, A/B traffic split, Slack alerts on regression.

Contributing

PRs welcome. See AGENTS.md if you're an AI coding agent working on this repo.

License

MIT — see LICENSE.