This repo runs an eval for a customer-support email classifier, diffs
the result against a baseline run record, and gates a prompt change
based on whether the diff is ok, warning, or critical. It is
the regression-detection layer for a single LLM feature; the feature
itself, the prompt versions, and the golden dataset all live in this
same repo. The eval pipeline uses Google's Gemini 2.5 Flash; the
regression detector is pure Python and is fully unit-testable offline.
If you are joining the team and need to add cases to the dataset, tune the detector, or run an eval against a new prompt, this README is the place to start.
- What is this
- Repository layout
- Setup
- Running the eval
- Adding cases to the golden dataset
- Adjusting thresholds
- How the regression detector works
- How the drift detector works
- CI and the merge gate
- Container image
- Slack alerts
- Web UI
- Chatbot
- Deploying the web UI
- Architecture decisions
- Dashboard
- Open questions
The feature under test is classify_email(email: str) -> Classification
in classifier.py. Given a customer email, it
returns a category (billing / technical / account / general)
and a one-sentence summary. Behavior is fully determined by the email
text plus a versioned prompt loaded from prompts/. The model is a
parameter, not a constant, so the same eval pipeline sweeps prompt
versions and model versions with no code change.
The eval pipeline is in eval/runner.py. It loads
the prompt, classifies every case in the golden dataset
(data/golden_dataset_v0.2.0.json),
embeds the generated summaries for cosine-similarity scoring, and
writes a timestamped run record to runs/. The regression detector
in eval/regression.py takes two run records
and produces a four-axis diff with a worst-of overall verdict. The
HTML report (eval/report.py) renders that diff
as a single self-contained file. The Slack alerter
(eval/alert.py) sends the diff to a configured
channel. The drift detector (eval/drift.py) flags
slow deterioration that no single run catches.
model-regression-detection/
├── prompts/ # Versioned prompts (v1, v2, v3, ...). One YAML per version.
├── data/ # Golden dataset + its loader/validator.
├── eval/
│ ├── runner.py # Classifies every case, writes a run record.
│ ├── regression.py # Diffs two run records; four axes, worst-of verdict.
│ ├── thresholds.yaml # Externalized threshold config (the only knob to tune).
│ ├── report.py # Renders a self-contained HTML diff report.
│ ├── alert.py # Posts a Slack Block Kit message.
│ ├── drift.py # Rolling-N-run drift detector.
│ └── run_eval.py # One-shot orchestrator: eval + diff + report + alert + drift.
├── webui/ # Flask demo app (the live web surface).
│ ├── app.py # Routes: /, /classify, /cases, /runs, /diff, /drift, /chat
│ ├── templates/ # Jinja templates
│ └── static/ # Vanilla CSS
├── chatbot/ # Function-calling assistant over the eval system.
│ ├── tools.py # Typed functions: get_latest_run, compare_two_runs, etc.
│ └── router.py # Heuristic + LLM routing
├── runs/ # Committed run records + HTML reports.
│ ├── BASELINE_v1_*.json # Pin these in the README when they're stable.
│ ├── CANDIDATE_v*_*.json
│ └── REPORT_*.html
├── classifier.py # The feature under test.
├── tests/ # 77 offline tests, no API keys, no network.
├── .github/workflows/eval.yml
├── Dockerfile # Multi-target: eval (default) or webui
├── docker-compose.yml
├── render.yaml # Render.com Blueprint for the web UI
├── Procfile # Heroku-style fallback
├── Makefile
└── README.md
Requires Python 3.12+ and a Gemini API key. Local development is the primary mode; Docker is for CI and reproducible runs.
git clone https://github.com/Sujay1709/Model-Regression-Detection-System.git
cd model-regression-detection
# Local venv
make install # or: python3 -m venv .venv && .venv/bin/pip install -r requirements.txt -r requirements-dev.txt
# API key
cp .env.example .env
# Edit .env to set GEMINI_API_KEY=...
# Optionally set SLACK_WEBHOOK_URL=https://hooks.slack.com/services/...
# Sanity check
make test # 33 offline tests
.venv/bin/python classifier.py # smoke test on a single emailFor containerized runs, see Container image.
The full pipeline (eval + diff + report + Slack + drift) is one
command. The orchestrator writes a new run record and a new HTML
report to runs/, posts to Slack if SLACK_WEBHOOK_URL is set, and
exits with the right code for CI.
.venv/bin/python -m eval.run_eval \
--prompt v3 --model gemini-2.5-flash \
--dataset data/golden_dataset_v0.2.0.json \
--baseline runs/BASELINE_v1_2026-07-07.jsonUseful flags:
--skip-eval— reuse the most recent run record for the given(prompt, model)instead of calling the API. Useful for re-running the diff against a new baseline without spending API calls.--no-alert— skip the Slack post. Useful for local dev so you don't spam a real channel.--no-drift— skip the drift log update.--report-url https://...— set the URL the Slack button links to. Defaults to the localruns/REPORT_*.htmlfile path.
The exit codes are part of the contract:
| code | meaning |
|---|---|
| 0 | ok — no significant change |
| 10 | warning — soft signal; review recommended |
| 20 | critical — hard regression; merge gate fails |
| 2 | insufficient_data — n_cases below min_cases_for_alert |
| 3 | error — missing baseline, missing dataset, etc. |
A typical CI gate is if [ $? -ge 20 ]; then exit 1; fi. A typical
notification gate is if [ $? -ge 10 ]; then notify "$SEVERITY"; fi.
The dataset is the source of truth for what the classifier should
do. Cases are hand-curated; we never populate them with LLM output.
The loader (data/dataset.py) validates every
field and refuses to load a malformed file.
To add cases:
-
Add new cases to the
casesarray, after the existing entries. Each case needs:id— stable, never reused; formatCASE-<4-digit-zero-padded-int>.input— verbatim customer email (preserve typos and casing).expected_category— one ofbilling,technical,account,general.ideal_summary— the one-sentence summary you want the model to produce. The eval embeds this and computes cosine similarity.expected_difficulty—easy/medium/hard.edge_case_tags— list; valid tags arenone,ambiguous,very_short,typos,mixed_language,sarcasm.notes— why this case matters. The detector doesn't read this, but a future engineer reading the dataset will.
-
Bump the version. The dataset file becomes
data/golden_dataset_v0.3.0.jsonanddataset_versionin the JSON is0.3.0. -
Validate:
.venv/bin/python -m data.dataset # lint the new file .venv/bin/python -m pytest tests/ -v # tests still pass
-
Run the eval against the new dataset:
.venv/bin/python eval/runner.py \ --prompt v3 --model gemini-2.5-flash \ --dataset data/golden_dataset_v0.3.0.json
A few design rules the dataset enforces, deliberately:
- One category per case. Multi-intent emails (a billing issue
and an account issue) get a single
expected_categoryand anotesfield explaining the tie-break. - Difficulty and edge tags are independent. An
easycase can still havesarcasmtag. Ahardcase can havenonetag. The detector uses both dimensions to slice results. - Cases are designed with counter-weights.
CASE-0006andCASE-0007look almost identical but expect different categories (cancel my subscription→ billing,delete my account→ account). These pairs are how the eval catches routing regressions.
The detector is operational, not statistical. Every threshold is in
eval/thresholds.yaml and the detector reads
it on every run. Tune the values to change sensitivity; no Python
change needed.
Thresholds are two-tier. A warn_* key is a soft signal (the
detector still returns a verdict, the Slack alert uses the warning
emoji, CI exits with 10). A critical_* key is a hard stop (CI
exits with 20, the alert uses the critical emoji, the merge gate
fails).
The current defaults and why we picked them:
| key | value | rationale |
|---|---|---|
warn_category_accuracy_drop |
0.03 | 3pp ≈ 2 cases at 75. Smallest delta with any confidence. |
critical_category_accuracy_drop |
0.08 | 8pp ≈ 6 cases at 75. Above run-to-run noise. |
warn_macro_f1_drop |
0.03 | Same as accuracy. Macro F1 is more honest across imbalanced categories. |
warn_latency_p95_increase |
1.30 | 30% slower. Free-tier noise is ~15-25% so 30% is the smallest signal. |
critical_latency_p95_increase |
2.00 | 2x slower. Almost always a real change. |
warn_slice_accuracy_drop |
0.08 | 8pp drop in any slice. The slice axis is the early warning amplifier. |
drift_window |
7 | Roughly 2 months of weekly runs. |
warn_drift_drop |
0.02 | 2pp drop in rolling 7-run avg. Catches 0.3pp/week drift. |
min_cases_for_alert |
5 | Below 5 cases, "the average" is just the most recent value. |
min_cases_for_slice_eval |
3 | Below 3 cases, one flipped row = 33pp noise. |
When the dataset grows past 200 cases, drop warn_* to 0.02 and
keep critical_* at 0.05. With a larger sample, the warn band can
be tighter without picking up noise.
The detector takes two run records (baseline and candidate) and
produces a RegressionReport with four independent axes:
- aggregate — overall category accuracy, macro F1, and summary similarity, with the candidate minus baseline delta on each. The verdict is the worst of the per-metric severity classifications.
- slices — per-difficulty and per-tag accuracy. The slice axis is the early warning for hidden regressions: aggregate can stay flat while one slice collapses.
- flips — which specific case IDs went from pass to fail or fail to pass. Three or more pass-to-fail flips with no offsetting fail-to-pass flips is a regression; the net improvement check requires a 2:1 asymmetry to upgrade an improvement verdict.
- latency — p50 / p95 latency and total token usage. Token cost is part of the contract now: a prompt that costs 2x more tokens for a 1pp accuracy gain is not a free lunch.
The overall verdict is the worst-of: regression on any one axis makes the run a regression. The overall severity is the worst-of the per-axis severities (ok / warning / critical). Improvements on one axis never mask regressions elsewhere.
Two design choices worth knowing about:
- Thresholds are not statistical tests. We use operational thresholds, not chi-squared or McNemar's. At 15-100 cases, the statistical tests lack the power to detect a 5pp shift reliably — they would give false confidence. The right question at this dataset size is "did this run look like the last run?", not "is this difference significant at p<0.05?".
- The detector is a pure data transform. No LLM calls, no network. That's why the test suite can verify its math offline against hand-built run records with known deltas.
The drift detector (eval/drift.py) looks at a
rolling window of N runs (default 7) and fires when the average
category accuracy drops by more than the configured threshold vs the
oldest entry in the window. The point is to catch gradual
deterioration: a prompt that drifts by 0.3pp per week for 10 weeks
won't trip any single-run alert, but the cumulative 3pp drop is
real and easy to miss in a sea of no_significant_change verdicts.
The drift log lives at eval/drift_log.json.
It's append-only and idempotent on (timestamp, prompt, model),
so CI re-runs don't double-record. The orchestrator appends on
every run. With a weekly cadence and a 7-run window, you get a
"slow drift" verdict that fires roughly every two months if a
prompt is genuinely drifting.
The drift signal is independent of the per-run verdict. A run can pass the per-run check but still fail the drift check; the combined verdict in the orchestrator is the worst of the two.
The eval runs on every PR that touches a relevant file (the prompt
directory, the dataset, the detector modules, or this workflow).
The workflow is in .github/workflows/eval.yml.
The job:
- Runs the offline test suite. If the detector is broken, the PR fails before the eval even runs.
- Determines the prompt version to test from the PR title. The
convention is
prompt: vN - <description>. If the title is silent, it defaults tov3. - Runs the eval against the golden dataset using the PR's prompt
changes. Reuses the committed v1 baseline at
runs/BASELINE_v1_2026-07-07.json. - Posts a summary comment on the PR. The comment is updated in place across pushes (no spam). The full HTML report is uploaded as a workflow artifact, downloadable from the PR's checks page.
- Fails the check if the combined severity is
critical. Warnings post the comment but allow the merge; the comment is the review record.
Required secrets (configured in the repo's GitHub Settings → Secrets and variables → Actions):
GEMINI_API_KEY— the same key you use locally. The free tier is enough for one eval per PR.SLACK_WEBHOOK_URL— optional. If unset, the orchestrator skips the Slack post (it's a no-op, not an error). The PR comment is the notification for the PR context.
A multi-stage Dockerfile packages the eval pipeline into a slim
image. The default command runs python -m eval.run_eval against
the v1 baseline. Override with docker run ... <args> to run any
of the other entry points.
# Build
docker build -t model-regression-detection:local .
# Run with env vars from .env
docker run --rm --env-file .env \
-v "$PWD/runs:/app/runs" \
model-regression-detection:local
# Or use docker-compose for the same thing with a cleaner CLI
docker compose run --rm evalThe env var contract:
| variable | required | purpose |
|---|---|---|
GEMINI_API_KEY |
yes | Gemini API key |
SLACK_WEBHOOK_URL |
no | Slack incoming webhook (orchestrator skips if unset) |
DRIFT_LOG_PATH |
no | Override the default eval/drift_log.json location |
THRESHOLDS_PATH |
no | Override the default eval/thresholds.yaml location |
BASELINE_RUN |
no | Override the default baseline filename |
The alerter (eval/alert.py) posts a Block Kit
message when the orchestrator finishes. The message is structured
for an on-call reader: severity emoji in the header, a color bar
on the left, headline deltas as a 2x2 grid, the cases that flipped
as a bulleted list, the reasons, and a button linking to the full
HTML report.
The alerter is a no-op when SLACK_WEBHOOK_URL is unset, so the
rest of the pipeline runs unchanged in environments without a
webhook configured (CI without Slack, local dev, unit tests).
To preview the message without sending:
.venv/bin/python eval/alert.py \
--baseline runs/BASELINE_v1_2026-07-07.json \
--candidate runs/CANDIDATE_v3_2026-07-08.json \
--report runs/REPORT_v1_vs_v3.html --dry-runA small Flask app at webui/ is the demo surface for the system.
A non-engineer can classify emails and browse the eval system in a
browser without cloning the repo or reading the code. The web UI
talks to the same classify_email function, the same data
loaders, and the same compare_runs the CLI uses. There is no
separate code path.
pip install -r requirements-webui.txt
make webui # serves on http://localhost:5000
make webui-prod # gunicorn on port 5000Routes:
| path | purpose |
|---|---|
GET / |
Landing page with the classifier form |
POST /classify |
Classify a pasted email |
GET /cases |
Browse the 75-case golden dataset |
GET /cases/<id> |
Inspect a single case (input, expected, notes) |
GET /runs |
List run records on disk |
GET /diff |
v1 baseline vs latest run, same engine the CLI uses |
GET /drift |
Rolling-N-run drift signal |
GET /report |
Latest HTML diff report |
GET /chat |
Chatbot UI shell |
POST /chat |
Chatbot Q&A endpoint (JSON in, JSON out) |
GET /healthz |
Liveness probe for deploy platforms |
The web UI is optional. The eval pipeline and the CI workflow do
not require it. The tests/test_webui.py suite exercises the
Flask routes offline using the test client; it skips cleanly when
Flask isn't installed.
A function-calling assistant at chatbot/ lets you ask the eval
system questions in plain English. It is a CLI today; the same
function backs the web UI's /chat endpoint.
.venv/bin/python -m chatbot "latest run"
.venv/bin/python -m chatbot "compare v1 and v3"
.venv/bin/python -m chatbot "CASE-0056"
.venv/bin/python -m chatbot "drift status"
.venv/bin/python -m chatbot --json "list runs"The router has two modes. The default is heuristic: a small
table of regex patterns maps questions to one of eight typed
functions in chatbot/tools.py. Heuristic routing is fast, free,
and deterministic; it covers every question a real user is
likely to ask. The fallback is LLM routing: with
GEMINI_API_KEY set, the router asks Gemini to pick the tool
and extract the arguments. The LLM path is the escape hatch for
the long tail ("which of the v2 sarcasm cases regressed in v3?")
and is invoked with --llm on the CLI.
The eight available tools, locked in TOOL_REGISTRY:
| tool | what it returns |
|---|---|
get_latest_run |
Headline numbers for the most recent run |
get_run_summary |
Same, scoped to a prompt version |
list_runs |
The N most recent runs |
compare_two_runs |
v1 vs v3 by default; configurable |
list_regressed_cases |
The case IDs that flipped pass→fail |
explain_case_failure |
Row-level detail for a specific case |
get_drift_status |
The current rolling-N-run drift signal |
get_dataset_case |
The golden-dataset definition for a case |
Every answer is a real query against the system's data, not a hallucination. That property is what makes the chatbot trustworthy in production: a question always maps to a function call, and the function returns the same answer the CLI would.
The web UI ships ready to deploy. Two options.
# 1. Sign in to https://dashboard.render.com with this repo's GitHub.
# 2. Click "New" -> "Blueprint" -> select this repo. Render reads
# render.yaml and creates the web service.
# 3. In the Render dashboard, set GEMINI_API_KEY and (optionally)
# SLACK_WEBHOOK_URL. Free tier spins down after 15 min of
# inactivity; first request after quiet takes ~30s.The render.yaml Blueprint builds the webui target of the
Dockerfile and deploys with gunicorn on port 10000. The health
check hits /healthz; failed checks trigger an automatic
restart.
docker build --build-arg TARGET=webui -t mrd:webui .
docker run --rm -p 5000:10000 -e GEMINI_API_KEY=... mrd:webui
# or use the local-dev gunicorn
make webui-prodThe Dockerfile supports two build targets (eval, default, and
webui) via the TARGET build-arg. The same image, with the
default target, runs the one-shot eval job; with TARGET=webui,
it serves the Flask app.
A few decisions that weren't obvious and are worth recording so the next person doesn't re-litigate them.
Why no async batching on the LLM calls? The free-tier Gemini API caps at 15 RPM. Async batching doesn't speed anything up at that ceiling; it just hits the rate limit faster and creates more concurrent 429 retries. Sequential calls with exponential backoff on 429 are the right architecture for this throughput. Async becomes meaningful at 100+ RPM on a paid tier.
Why operational thresholds instead of statistical tests? At
15-100 cases, chi-squared and McNemar's lack the power to detect
a 5pp shift reliably. The operational thresholds in
eval/thresholds.yaml are the values a practitioner would set
from experience with the dataset. The point of the detector is
"did this run look like the last run?", not "is this difference
significant at p<0.05?". With a 200+ case dataset, this can be
revisited.
Why two severity tiers (warn/critical) instead of one? The on-call reader needs to distinguish "this is worth a look" from "this is definitely broken". The CI gate can enforce one and notify on the other. A single threshold conflates the two signals and trains people to ignore the alert.
Why is the regression detector a pure data transform? No LLM calls, no network. That's why the test suite verifies its math offline against hand-built run records with known deltas. The detector is the part of the system we cannot afford to be wrong about; testing it offline is the highest-value test coverage we can have.
Why an externalized threshold config? Tunes by PM, on-call
engineer, or domain expert don't require code changes. The
detector reads eval/thresholds.yaml on every run, so a config
change takes effect on the next eval — no redeploy, no rebuild.
Why commit the run records to the repo? Reproducibility. The detector is deterministic given the same inputs; a committed run record lets any future engineer reproduce the diff that produced a given verdict. It also makes the eval system self-describing: the repo contains the prompts, the dataset, the detector, and the data that produced the headlines in the README.
Why a multi-stage Docker build? The Python SDK + numpy + yaml
is around 250MB installed. The multi-stage build produces a
runtime image around 180MB. We avoid python:3.12-alpine because
numpy and the Google SDK don't ship musllinux wheels; alpine would
force a source build of numpy, adding 5+ minutes to the build.
Why run as a non-root user inside the container? Defense in
depth. The eval writes to /app/runs/, which the non-root user
owns. If a dependency ever exploits a path traversal, the blast
radius is contained.
The rest of this README is internal docs. What follows is a short, externally-readable summary of the project — the kind of thing that would live on a personal blog or the GitHub project page. Three paragraphs, one design decision.
Teams ship prompt changes blind. Someone tweaks a prompt to fix a routing bug, the new prompt is merged, and the change is discovered three days later when a customer complains that the chatbot stopped understanding password reset emails. There is no equivalent of a unit test for "the model still routes the same way it did yesterday." The eval system we have for offline benchmarks doesn't catch per-prompt regressions, and the production observability stack is too noisy to spot a 5-point drop in sarcasm accuracy.
Treat prompt changes like code changes. Every PR that touches
the prompt directory runs an eval against a 75-case hand-curated
golden dataset, generates an HTML diff report, posts a summary
comment to the PR, and blocks the merge if the new prompt
introduces a critical regression. The eval uses the same
classifier, the same data, and the same model the user is
running. The diff is computed by a pure-data detector with two
severity tiers: warning (a soft signal, posted as a comment
but the merge proceeds) and critical (a hard stop, the merge
fails). CI exit codes match the severity: 0, 10, 20. The
detector is unit-testable offline against hand-built run records
with known deltas — 77 fixture-based tests, no API calls, no
network.
The detector also tracks drift — a rolling 7-run average of
category accuracy — separately from per-run regressions. Per-run
detection asks "did this run look like the last run?". Drift
detection asks "is the prompt slowly degrading across many
runs?". A prompt that drifts by 0.3 percentage points per week
for ten weeks won't trip any single-run alert, but the
cumulative 3-point drop is real, important, and easy to miss in
a sea of "no significant change" verdicts. By separating the two
signals, the system catches both the loud failures (a single
prompt change that broke 6 cases) and the slow ones (a prompt
that was fine on day 1 and is at 90% on day 70). The
implementation is in eval/drift.py — 200 lines, a rolling
log at eval/drift_log.json, and a min_runs_for_drift floor
that waits for enough data to fire. It's the part of the system
I'm proudest of because it's the part that does the thing no
single test or single eval can do.
- LLM-as-judge. The detector currently scores summaries by cosine similarity to a hand-written reference. An LLM judge would be a richer signal but doubles the API cost per run and introduces a new regression surface (the judge can drift). Worth adding as an optional enrichment pass on runs you care about, not as a default.
- Cross-model benchmarks. The eval pipeline already supports
--model <anything>. A real benchmark of Gemini 2.5 Flash vs Claude vs GPT-4o-mini on the 75-case dataset is a separate artifact, not a code change. - Dataset expansion past 100 cases. At 200+ cases the
statistical-power argument flips and chi-squared becomes
viable. The threshold config is the place to start; revisit
min_cases_for_alertand the per-metric warn/critical bands.
A Streamlit dashboard reads the same files the CLI does and renders
them as a web UI. It is read-only: the dashboard never writes to
runs/, never posts to Slack, never calls the LLM. To run an eval,
use make eval. To browse runs, use the dashboard.
pip install -r requirements-dashboard.txt
make dashboard # serves on http://localhost:8501Pages:
- Overview — headline numbers for the latest run, the v1 -> latest diff (if both exist), dataset summary, drift summary.
- Runs — the timeline of every run record with sortable columns, a dropdown to drill into any single run, plus the per-slice table and confusion matrix from the run record.
- Diff — pick any two runs from the dropdowns, get the same four-axis diff the CLI produces, plus the regressed-cases table and the reasons.
- Drift — the rolling-N-run accuracy chart over time, with a table of the raw drift log entries.
- Dataset — the golden dataset with filters by category, difficulty, and edge tag, plus a per-case detail expander.
The dashboard is optional. The eval pipeline and the CI workflow
do not require it. The tests/test_dashboard_smoke.py suite
exercises the dashboard's data layer offline so the contract
between the dashboard and the rest of the system is locked.
MIT. See LICENSE.