A multi-agent data-analytics assistant over the NYC 311 service-request dataset. Ask questions in plain English, get statistics with the SQL that produced them, the rows they came from, and a check that every number in the answer exists in those rows. When the data can't answer a question, it says so instead of guessing.
Stack: DuckDB · LangGraph (6 agents) · DeepSeek / any OpenAI-compatible model · FastAPI · React + Vite · Docker.
backend/.env # add LLM_API_KEY, LLM_MODEL, a real SESSION_SECRET
# put the dataset CSV in backend/data/ (or set SAMPLE_DATA: "1" in docker-compose.yml to use synthetic data)
docker compose up --buildOpen http://localhost:8000. First start ingests the CSV into DuckDB (a minute or two); later starts are instant.
Details, dev override and Windows notes: docs/DOCKER.md.
# backend
cd backend
python -m venv .venv && .venv\Scripts\activate # macOS/Linux: source .venv/bin/activate
pip install -r requirements.txt
copy .env.example .env # set LLM_API_KEY, LLM_MODEL
python scripts/ingest.py --min-date 2015-01-01 # CSV -> data/nyc311.duckdb
python scripts/profile.py # -> data/profile.md (data report)
uvicorn app.main:app --reload --port 8000
# frontend (second terminal)
cd frontend
npm install
npm run dev # http://localhost:5173, proxies /api -> :8000No dataset at hand? python scripts/make_sample_data.py writes a synthetic CSV with the same schema.
No API credits? Set LLM_MODEL=mock — the whole stack runs offline (the mock answers the golden questions).
cd backend
python -m pytest # 98 offline tests: guard, parsing, grounding, truth SQL, viz, reviewer, API, full graph
python -m evals.run_eval --mock # eval harness plumbing, offline
python -m evals.run_eval # real accuracy numbers with your model -> evals/reports/user ─► Router ─► SQL Analyst ─► Reviewer ─► Narrator ─► Viz ─► answer
│ │ ▲ │ rejected once → back to SQL Analyst
│ └──┘ guard + retry (max 3)
├─► Meta ("what data do you have?")
├─► Viz directly ("show that as a bar chart" – re-uses the last table)
└─► reply directly: out of scope · needs clarification · small talk
| agent | job | what keeps it honest |
|---|---|---|
| Router | classify the message; rewrite follow-ups ("now only Brooklyn") into standalone questions | sees only the live coverage block; refusing is a normal output |
| SQL Analyst | question → one DuckDB SELECT as a structured plan (query / out_of_scope / clarify) |
sql_guard.py (sqlglot AST): single read-only SELECT, one table, every column must exist; errors fed back for retry |
| Reviewer | does this SQL answer this question? | rule checks from the semantic layer (denominator trap, raw vs clean columns, median vs avg, % range) that the model cannot override, then an LLM judgement; one send-back, then proceed with concerns attached |
| Narrator | write the answer from the result table | sees only the rows; every number in the text is checked against the table; two strikes → deterministic template |
| Viz | suggest a chart (opt-in in the UI; bar, line, scatter, pie) | may only choose among (type, x, y) combinations that already pass validate_spec (pie only for parts of a whole); invalid choice → rule-based pick |
| Meta | describe the dataset | answers from computed coverage, not memory |
Dataset understanding lives in two places: backend/agent/schema.md (hand-written semantic layer: column
meanings, exact metric definitions, forbidden columns, scope rules) and backend/agent/coverage.py (computed at
startup from the live database: date range, years, value lists, quality percentages). Nothing about the dataset's
contents is hard-coded — swap the file and the agents' knowledge updates.
No fabrication is enforced by code, not prompts: unknown columns die in the guard; invented numbers die in the grounding check; invented chart columns die in the chart validator; out-of-range dates and unknown categories are refused with a statement of what the data does cover; provider failures become one clear user-facing message.
Current result (gpt-4o-mini, 25-case golden set): 25/25 correct, 0 fabricated numbers, 5/5 correct refusals;
Reviewer: 0 % false rejects, 88.6 % of injected SQL errors caught. Three tuning rounds took it from 68 % → 88 % →
100 %; the full record, including the two incidents that shaped the design, is in docs/EVALS.md.
backend/evals/golden.jsonl — 25 cases: aggregates, percentages, CTE/top-N, time, geo, family filters,
5 out-of-scope, 1 ambiguous, meta, chitchat, 2 multi-turn follow-ups. Ground truth is a truth_sql executed live
against the same database; agent results are compared by value (any equivalent SQL passes).
Per case: intent · SQL executed · answer matches truth (tolerance) · every narrated number grounded · chart spec valid
· follow-up rewritten. Per run: pass rate by tag with ▲▼ vs the previous report, SQL attempts, narrator fallbacks,
reviewer send-backs. tests/test_llm_live.py is the real-model gate (refusals 100 %, overall ≥ 85 %).
Judge eval (python -m evals.judge_eval) measures the Reviewer itself: every truth query must be approved
(false-reject rate) and six kinds of deliberately broken variants — dropped filter, wrong LIMIT, avg-for-median,
inverted ordering, raw columns, the denominator trap — must be rejected (catch rate per mutation, and by which layer:
rules or model). Rule-covered mutations are caught 100 % deterministically; the report shows what only the model catches.
The ground truth is itself tested (test_golden_truth.py): every truth query must pass the guard, run, and avoid the
avg((x<=N)::INT) denominator bug we found while building this.
backend/
agent/ schema.md (semantic layer) · coverage.py · sql_guard.py · sql_agent.py · graph.py · state.py
nodes/ router.py · sql_analyst.py · reviewer.py · narrator.py · viz.py
app/ FastAPI: main.py · service.py · store.py (SQLite threads) · schemas.py · errors.py · config.py
evals/ golden.jsonl · scorers.py · run_eval.py · mock_llm.py
scripts/ ingest.py · profile.py · ask.py · chat.py · make_sample_data.py · entrypoint.sh
tests/ 98 offline tests
frontend/ React + TypeScript (Vite): accounts, conversation history sidebar, streaming chat, agent trail, reasoning, evidence, charts
docs/api.md contract · docs/AUTH.md sign-in & history · docs/EVALS.md evaluation record · docs/DEPLOY.md · docs/BUILD_LOG.md
Dockerfile · docker-compose.yml · .github/workflows/ci.yml
| var | meaning |
|---|---|
LLM_API_KEY, LLM_MODEL, LLM_BASE_URL |
any OpenAI-compatible provider; LLM_MODEL=mock runs offline |
LLM_TEMPERATURE, LLM_JSON_MODE |
0/empty; JSON response format on/off |
REVIEWER_LLM |
0 skips the Reviewer's model call (rules still run) |
DUCKDB_PATH, RAW_CSV_PATH, MIN_CREATED_DATE, THREADS_DB_PATH |
data locations |
CORS_ORIGINS, API_PORT, LOG_LEVEL, STATIC_DIR |
server |
API_TOKEN, RATE_LIMIT_PER_MINUTE, DAILY_QUESTION_BUDGET, MAX_TURNS_PER_THREAD |
guardrails (see below) |
APP_TITLE |
what the UI calls itself; the UI shows no dataset details until asked |
AUTH_MODE, SESSION_SECRET, SESSION_DAYS, COOKIE_SECURE, LOGIN_ATTEMPTS_PER_MINUTE |
accounts (email + password) and per-user conversation history (see docs/AUTH.md) |
Data safety: the DuckDB connection is opened read-only; the SQL guard parses every query with sqlglot and allows
one SELECT over one table with only existing columns, blocks file/environment functions, and adds a row cap; execution
has a 20 s timeout and a 500-row limit. Model safety: closed output schemas per agent, number grounding on every
answer, chart columns validated, deterministic Reviewer rules the model cannot override, data cells sanitised and
framed as data before entering any prompt. Service safety: optional bearer token, per-IP rate limit, daily budget,
per-thread turn cap, user-facing error messages with details only in server logs.
The provided extract contains 364,558 NYPD-handled requests from calendar year 2015. The agents treat any question outside that scope (other years, other agencies' complaint types such as heating or rodents, external data such as population) as unanswerable and say so. Coverage is computed from the file, so a broader extract needs no code changes.
License: MIT