Skip to content

Repository files navigation

model-regression-detection

This repo runs an eval for a customer-support email classifier, diffs the result against a baseline run record, and gates a prompt change based on whether the diff is ok, warning, or critical. It is the regression-detection layer for a single LLM feature; the feature itself, the prompt versions, and the golden dataset all live in this same repo. The eval pipeline uses Google's Gemini 2.5 Flash; the regression detector is pure Python and is fully unit-testable offline.

If you are joining the team and need to add cases to the dataset, tune the detector, or run an eval against a new prompt, this README is the place to start.

Table of contents

What is this

The feature under test is classify_email(email: str) -> Classification in classifier.py. Given a customer email, it returns a category (billing / technical / account / general) and a one-sentence summary. Behavior is fully determined by the email text plus a versioned prompt loaded from prompts/. The model is a parameter, not a constant, so the same eval pipeline sweeps prompt versions and model versions with no code change.

The eval pipeline is in eval/runner.py. It loads the prompt, classifies every case in the golden dataset (data/golden_dataset_v0.2.0.json), embeds the generated summaries for cosine-similarity scoring, and writes a timestamped run record to runs/. The regression detector in eval/regression.py takes two run records and produces a four-axis diff with a worst-of overall verdict. The HTML report (eval/report.py) renders that diff as a single self-contained file. The Slack alerter (eval/alert.py) sends the diff to a configured channel. The drift detector (eval/drift.py) flags slow deterioration that no single run catches.

Repository layout

model-regression-detection/
├── prompts/                  # Versioned prompts (v1, v2, v3, ...). One YAML per version.
├── data/                     # Golden dataset + its loader/validator.
├── eval/
│   ├── runner.py             # Classifies every case, writes a run record.
│   ├── regression.py         # Diffs two run records; four axes, worst-of verdict.
│   ├── thresholds.yaml       # Externalized threshold config (the only knob to tune).
│   ├── report.py             # Renders a self-contained HTML diff report.
│   ├── alert.py              # Posts a Slack Block Kit message.
│   ├── drift.py              # Rolling-N-run drift detector.
│   └── run_eval.py           # One-shot orchestrator: eval + diff + report + alert + drift.
├── webui/                    # Flask demo app (the live web surface).
│   ├── app.py                # Routes: /, /classify, /cases, /runs, /diff, /drift, /chat
│   ├── templates/            # Jinja templates
│   └── static/               # Vanilla CSS
├── chatbot/                  # Function-calling assistant over the eval system.
│   ├── tools.py              # Typed functions: get_latest_run, compare_two_runs, etc.
│   └── router.py             # Heuristic + LLM routing
├── runs/                     # Committed run records + HTML reports.
│   ├── BASELINE_v1_*.json    # Pin these in the README when they're stable.
│   ├── CANDIDATE_v*_*.json
│   └── REPORT_*.html
├── classifier.py             # The feature under test.
├── tests/                    # 77 offline tests, no API keys, no network.
├── .github/workflows/eval.yml
├── Dockerfile                # Multi-target: eval (default) or webui
├── docker-compose.yml
├── render.yaml               # Render.com Blueprint for the web UI
├── Procfile                  # Heroku-style fallback
├── Makefile
└── README.md

Setup

Requires Python 3.12+ and a Gemini API key. Local development is the primary mode; Docker is for CI and reproducible runs.

git clone https://github.com/Sujay1709/Model-Regression-Detection-System.git
cd model-regression-detection

# Local venv
make install                  # or: python3 -m venv .venv && .venv/bin/pip install -r requirements.txt -r requirements-dev.txt

# API key
cp .env.example .env
# Edit .env to set GEMINI_API_KEY=...
# Optionally set SLACK_WEBHOOK_URL=https://hooks.slack.com/services/...

# Sanity check
make test                     # 33 offline tests
.venv/bin/python classifier.py # smoke test on a single email

For containerized runs, see Container image.

Running the eval

The full pipeline (eval + diff + report + Slack + drift) is one command. The orchestrator writes a new run record and a new HTML report to runs/, posts to Slack if SLACK_WEBHOOK_URL is set, and exits with the right code for CI.

.venv/bin/python -m eval.run_eval \
    --prompt v3 --model gemini-2.5-flash \
    --dataset data/golden_dataset_v0.2.0.json \
    --baseline runs/BASELINE_v1_2026-07-07.json

Useful flags:

  • --skip-eval — reuse the most recent run record for the given (prompt, model) instead of calling the API. Useful for re-running the diff against a new baseline without spending API calls.
  • --no-alert — skip the Slack post. Useful for local dev so you don't spam a real channel.
  • --no-drift — skip the drift log update.
  • --report-url https://... — set the URL the Slack button links to. Defaults to the local runs/REPORT_*.html file path.

The exit codes are part of the contract:

code meaning
0 ok — no significant change
10 warning — soft signal; review recommended
20 critical — hard regression; merge gate fails
2 insufficient_data — n_cases below min_cases_for_alert
3 error — missing baseline, missing dataset, etc.

A typical CI gate is if [ $? -ge 20 ]; then exit 1; fi. A typical notification gate is if [ $? -ge 10 ]; then notify "$SEVERITY"; fi.

Adding cases to the golden dataset

The dataset is the source of truth for what the classifier should do. Cases are hand-curated; we never populate them with LLM output. The loader (data/dataset.py) validates every field and refuses to load a malformed file.

To add cases:

  1. Open data/golden_dataset_v0.2.0.json.

  2. Add new cases to the cases array, after the existing entries. Each case needs:

    • id — stable, never reused; format CASE-<4-digit-zero-padded-int>.
    • input — verbatim customer email (preserve typos and casing).
    • expected_category — one of billing, technical, account, general.
    • ideal_summary — the one-sentence summary you want the model to produce. The eval embeds this and computes cosine similarity.
    • expected_difficultyeasy / medium / hard.
    • edge_case_tags — list; valid tags are none, ambiguous, very_short, typos, mixed_language, sarcasm.
    • notes — why this case matters. The detector doesn't read this, but a future engineer reading the dataset will.
  3. Bump the version. The dataset file becomes data/golden_dataset_v0.3.0.json and dataset_version in the JSON is 0.3.0.

  4. Validate:

    .venv/bin/python -m data.dataset        # lint the new file
    .venv/bin/python -m pytest tests/ -v    # tests still pass
  5. Run the eval against the new dataset:

    .venv/bin/python eval/runner.py \
        --prompt v3 --model gemini-2.5-flash \
        --dataset data/golden_dataset_v0.3.0.json

A few design rules the dataset enforces, deliberately:

  • One category per case. Multi-intent emails (a billing issue and an account issue) get a single expected_category and a notes field explaining the tie-break.
  • Difficulty and edge tags are independent. An easy case can still have sarcasm tag. A hard case can have none tag. The detector uses both dimensions to slice results.
  • Cases are designed with counter-weights. CASE-0006 and CASE-0007 look almost identical but expect different categories (cancel my subscription → billing, delete my account → account). These pairs are how the eval catches routing regressions.

Adjusting thresholds

The detector is operational, not statistical. Every threshold is in eval/thresholds.yaml and the detector reads it on every run. Tune the values to change sensitivity; no Python change needed.

Thresholds are two-tier. A warn_* key is a soft signal (the detector still returns a verdict, the Slack alert uses the warning emoji, CI exits with 10). A critical_* key is a hard stop (CI exits with 20, the alert uses the critical emoji, the merge gate fails).

The current defaults and why we picked them:

key value rationale
warn_category_accuracy_drop 0.03 3pp ≈ 2 cases at 75. Smallest delta with any confidence.
critical_category_accuracy_drop 0.08 8pp ≈ 6 cases at 75. Above run-to-run noise.
warn_macro_f1_drop 0.03 Same as accuracy. Macro F1 is more honest across imbalanced categories.
warn_latency_p95_increase 1.30 30% slower. Free-tier noise is ~15-25% so 30% is the smallest signal.
critical_latency_p95_increase 2.00 2x slower. Almost always a real change.
warn_slice_accuracy_drop 0.08 8pp drop in any slice. The slice axis is the early warning amplifier.
drift_window 7 Roughly 2 months of weekly runs.
warn_drift_drop 0.02 2pp drop in rolling 7-run avg. Catches 0.3pp/week drift.
min_cases_for_alert 5 Below 5 cases, "the average" is just the most recent value.
min_cases_for_slice_eval 3 Below 3 cases, one flipped row = 33pp noise.

When the dataset grows past 200 cases, drop warn_* to 0.02 and keep critical_* at 0.05. With a larger sample, the warn band can be tighter without picking up noise.

How the regression detector works

The detector takes two run records (baseline and candidate) and produces a RegressionReport with four independent axes:

  • aggregate — overall category accuracy, macro F1, and summary similarity, with the candidate minus baseline delta on each. The verdict is the worst of the per-metric severity classifications.
  • slices — per-difficulty and per-tag accuracy. The slice axis is the early warning for hidden regressions: aggregate can stay flat while one slice collapses.
  • flips — which specific case IDs went from pass to fail or fail to pass. Three or more pass-to-fail flips with no offsetting fail-to-pass flips is a regression; the net improvement check requires a 2:1 asymmetry to upgrade an improvement verdict.
  • latency — p50 / p95 latency and total token usage. Token cost is part of the contract now: a prompt that costs 2x more tokens for a 1pp accuracy gain is not a free lunch.

The overall verdict is the worst-of: regression on any one axis makes the run a regression. The overall severity is the worst-of the per-axis severities (ok / warning / critical). Improvements on one axis never mask regressions elsewhere.

Two design choices worth knowing about:

  • Thresholds are not statistical tests. We use operational thresholds, not chi-squared or McNemar's. At 15-100 cases, the statistical tests lack the power to detect a 5pp shift reliably — they would give false confidence. The right question at this dataset size is "did this run look like the last run?", not "is this difference significant at p<0.05?".
  • The detector is a pure data transform. No LLM calls, no network. That's why the test suite can verify its math offline against hand-built run records with known deltas.

How the drift detector works

The drift detector (eval/drift.py) looks at a rolling window of N runs (default 7) and fires when the average category accuracy drops by more than the configured threshold vs the oldest entry in the window. The point is to catch gradual deterioration: a prompt that drifts by 0.3pp per week for 10 weeks won't trip any single-run alert, but the cumulative 3pp drop is real and easy to miss in a sea of no_significant_change verdicts.

The drift log lives at eval/drift_log.json. It's append-only and idempotent on (timestamp, prompt, model), so CI re-runs don't double-record. The orchestrator appends on every run. With a weekly cadence and a 7-run window, you get a "slow drift" verdict that fires roughly every two months if a prompt is genuinely drifting.

The drift signal is independent of the per-run verdict. A run can pass the per-run check but still fail the drift check; the combined verdict in the orchestrator is the worst of the two.

CI and the merge gate

The eval runs on every PR that touches a relevant file (the prompt directory, the dataset, the detector modules, or this workflow). The workflow is in .github/workflows/eval.yml.

The job:

  1. Runs the offline test suite. If the detector is broken, the PR fails before the eval even runs.
  2. Determines the prompt version to test from the PR title. The convention is prompt: vN - <description>. If the title is silent, it defaults to v3.
  3. Runs the eval against the golden dataset using the PR's prompt changes. Reuses the committed v1 baseline at runs/BASELINE_v1_2026-07-07.json.
  4. Posts a summary comment on the PR. The comment is updated in place across pushes (no spam). The full HTML report is uploaded as a workflow artifact, downloadable from the PR's checks page.
  5. Fails the check if the combined severity is critical. Warnings post the comment but allow the merge; the comment is the review record.

Required secrets (configured in the repo's GitHub Settings → Secrets and variables → Actions):

  • GEMINI_API_KEY — the same key you use locally. The free tier is enough for one eval per PR.
  • SLACK_WEBHOOK_URL — optional. If unset, the orchestrator skips the Slack post (it's a no-op, not an error). The PR comment is the notification for the PR context.

Container image

A multi-stage Dockerfile packages the eval pipeline into a slim image. The default command runs python -m eval.run_eval against the v1 baseline. Override with docker run ... <args> to run any of the other entry points.

# Build
docker build -t model-regression-detection:local .

# Run with env vars from .env
docker run --rm --env-file .env \
    -v "$PWD/runs:/app/runs" \
    model-regression-detection:local

# Or use docker-compose for the same thing with a cleaner CLI
docker compose run --rm eval

The env var contract:

variable required purpose
GEMINI_API_KEY yes Gemini API key
SLACK_WEBHOOK_URL no Slack incoming webhook (orchestrator skips if unset)
DRIFT_LOG_PATH no Override the default eval/drift_log.json location
THRESHOLDS_PATH no Override the default eval/thresholds.yaml location
BASELINE_RUN no Override the default baseline filename

Slack alerts

The alerter (eval/alert.py) posts a Block Kit message when the orchestrator finishes. The message is structured for an on-call reader: severity emoji in the header, a color bar on the left, headline deltas as a 2x2 grid, the cases that flipped as a bulleted list, the reasons, and a button linking to the full HTML report.

The alerter is a no-op when SLACK_WEBHOOK_URL is unset, so the rest of the pipeline runs unchanged in environments without a webhook configured (CI without Slack, local dev, unit tests).

To preview the message without sending:

.venv/bin/python eval/alert.py \
    --baseline runs/BASELINE_v1_2026-07-07.json \
    --candidate runs/CANDIDATE_v3_2026-07-08.json \
    --report runs/REPORT_v1_vs_v3.html --dry-run

Web UI

A small Flask app at webui/ is the demo surface for the system. A non-engineer can classify emails and browse the eval system in a browser without cloning the repo or reading the code. The web UI talks to the same classify_email function, the same data loaders, and the same compare_runs the CLI uses. There is no separate code path.

pip install -r requirements-webui.txt
make webui                   # serves on http://localhost:5000
make webui-prod              # gunicorn on port 5000

Routes:

path purpose
GET / Landing page with the classifier form
POST /classify Classify a pasted email
GET /cases Browse the 75-case golden dataset
GET /cases/<id> Inspect a single case (input, expected, notes)
GET /runs List run records on disk
GET /diff v1 baseline vs latest run, same engine the CLI uses
GET /drift Rolling-N-run drift signal
GET /report Latest HTML diff report
GET /chat Chatbot UI shell
POST /chat Chatbot Q&A endpoint (JSON in, JSON out)
GET /healthz Liveness probe for deploy platforms

The web UI is optional. The eval pipeline and the CI workflow do not require it. The tests/test_webui.py suite exercises the Flask routes offline using the test client; it skips cleanly when Flask isn't installed.

Chatbot

A function-calling assistant at chatbot/ lets you ask the eval system questions in plain English. It is a CLI today; the same function backs the web UI's /chat endpoint.

.venv/bin/python -m chatbot "latest run"
.venv/bin/python -m chatbot "compare v1 and v3"
.venv/bin/python -m chatbot "CASE-0056"
.venv/bin/python -m chatbot "drift status"
.venv/bin/python -m chatbot --json "list runs"

The router has two modes. The default is heuristic: a small table of regex patterns maps questions to one of eight typed functions in chatbot/tools.py. Heuristic routing is fast, free, and deterministic; it covers every question a real user is likely to ask. The fallback is LLM routing: with GEMINI_API_KEY set, the router asks Gemini to pick the tool and extract the arguments. The LLM path is the escape hatch for the long tail ("which of the v2 sarcasm cases regressed in v3?") and is invoked with --llm on the CLI.

The eight available tools, locked in TOOL_REGISTRY:

tool what it returns
get_latest_run Headline numbers for the most recent run
get_run_summary Same, scoped to a prompt version
list_runs The N most recent runs
compare_two_runs v1 vs v3 by default; configurable
list_regressed_cases The case IDs that flipped pass→fail
explain_case_failure Row-level detail for a specific case
get_drift_status The current rolling-N-run drift signal
get_dataset_case The golden-dataset definition for a case

Every answer is a real query against the system's data, not a hallucination. That property is what makes the chatbot trustworthy in production: a question always maps to a function call, and the function returns the same answer the CLI would.

Deploying the web UI

The web UI ships ready to deploy. Two options.

Render.com (recommended, free tier)

# 1. Sign in to https://dashboard.render.com with this repo's GitHub.
# 2. Click "New" -> "Blueprint" -> select this repo. Render reads
#    render.yaml and creates the web service.
# 3. In the Render dashboard, set GEMINI_API_KEY and (optionally)
#    SLACK_WEBHOOK_URL. Free tier spins down after 15 min of
#    inactivity; first request after quiet takes ~30s.

The render.yaml Blueprint builds the webui target of the Dockerfile and deploys with gunicorn on port 10000. The health check hits /healthz; failed checks trigger an automatic restart.

Any Docker host

docker build --build-arg TARGET=webui -t mrd:webui .
docker run --rm -p 5000:10000 -e GEMINI_API_KEY=... mrd:webui
# or use the local-dev gunicorn
make webui-prod

The Dockerfile supports two build targets (eval, default, and webui) via the TARGET build-arg. The same image, with the default target, runs the one-shot eval job; with TARGET=webui, it serves the Flask app.

Architecture decisions

A few decisions that weren't obvious and are worth recording so the next person doesn't re-litigate them.

Why no async batching on the LLM calls? The free-tier Gemini API caps at 15 RPM. Async batching doesn't speed anything up at that ceiling; it just hits the rate limit faster and creates more concurrent 429 retries. Sequential calls with exponential backoff on 429 are the right architecture for this throughput. Async becomes meaningful at 100+ RPM on a paid tier.

Why operational thresholds instead of statistical tests? At 15-100 cases, chi-squared and McNemar's lack the power to detect a 5pp shift reliably. The operational thresholds in eval/thresholds.yaml are the values a practitioner would set from experience with the dataset. The point of the detector is "did this run look like the last run?", not "is this difference significant at p<0.05?". With a 200+ case dataset, this can be revisited.

Why two severity tiers (warn/critical) instead of one? The on-call reader needs to distinguish "this is worth a look" from "this is definitely broken". The CI gate can enforce one and notify on the other. A single threshold conflates the two signals and trains people to ignore the alert.

Why is the regression detector a pure data transform? No LLM calls, no network. That's why the test suite verifies its math offline against hand-built run records with known deltas. The detector is the part of the system we cannot afford to be wrong about; testing it offline is the highest-value test coverage we can have.

Why an externalized threshold config? Tunes by PM, on-call engineer, or domain expert don't require code changes. The detector reads eval/thresholds.yaml on every run, so a config change takes effect on the next eval — no redeploy, no rebuild.

Why commit the run records to the repo? Reproducibility. The detector is deterministic given the same inputs; a committed run record lets any future engineer reproduce the diff that produced a given verdict. It also makes the eval system self-describing: the repo contains the prompts, the dataset, the detector, and the data that produced the headlines in the README.

Why a multi-stage Docker build? The Python SDK + numpy + yaml is around 250MB installed. The multi-stage build produces a runtime image around 180MB. We avoid python:3.12-alpine because numpy and the Google SDK don't ship musllinux wheels; alpine would force a source build of numpy, adding 5+ minutes to the build.

Why run as a non-root user inside the container? Defense in depth. The eval writes to /app/runs/, which the non-root user owns. If a dependency ever exploits a path traversal, the blast radius is contained.

Blog post: CI/CD for LLM behavior

The rest of this README is internal docs. What follows is a short, externally-readable summary of the project — the kind of thing that would live on a personal blog or the GitHub project page. Three paragraphs, one design decision.

The problem

Teams ship prompt changes blind. Someone tweaks a prompt to fix a routing bug, the new prompt is merged, and the change is discovered three days later when a customer complains that the chatbot stopped understanding password reset emails. There is no equivalent of a unit test for "the model still routes the same way it did yesterday." The eval system we have for offline benchmarks doesn't catch per-prompt regressions, and the production observability stack is too noisy to spot a 5-point drop in sarcasm accuracy.

The approach

Treat prompt changes like code changes. Every PR that touches the prompt directory runs an eval against a 75-case hand-curated golden dataset, generates an HTML diff report, posts a summary comment to the PR, and blocks the merge if the new prompt introduces a critical regression. The eval uses the same classifier, the same data, and the same model the user is running. The diff is computed by a pure-data detector with two severity tiers: warning (a soft signal, posted as a comment but the merge proceeds) and critical (a hard stop, the merge fails). CI exit codes match the severity: 0, 10, 20. The detector is unit-testable offline against hand-built run records with known deltas — 77 fixture-based tests, no API calls, no network.

The design decision I'm proud of

The detector also tracks drift — a rolling 7-run average of category accuracy — separately from per-run regressions. Per-run detection asks "did this run look like the last run?". Drift detection asks "is the prompt slowly degrading across many runs?". A prompt that drifts by 0.3 percentage points per week for ten weeks won't trip any single-run alert, but the cumulative 3-point drop is real, important, and easy to miss in a sea of "no significant change" verdicts. By separating the two signals, the system catches both the loud failures (a single prompt change that broke 6 cases) and the slow ones (a prompt that was fine on day 1 and is at 90% on day 70). The implementation is in eval/drift.py — 200 lines, a rolling log at eval/drift_log.json, and a min_runs_for_drift floor that waits for enough data to fire. It's the part of the system I'm proudest of because it's the part that does the thing no single test or single eval can do.

Open questions

  • LLM-as-judge. The detector currently scores summaries by cosine similarity to a hand-written reference. An LLM judge would be a richer signal but doubles the API cost per run and introduces a new regression surface (the judge can drift). Worth adding as an optional enrichment pass on runs you care about, not as a default.
  • Cross-model benchmarks. The eval pipeline already supports --model <anything>. A real benchmark of Gemini 2.5 Flash vs Claude vs GPT-4o-mini on the 75-case dataset is a separate artifact, not a code change.
  • Dataset expansion past 100 cases. At 200+ cases the statistical-power argument flips and chi-squared becomes viable. The threshold config is the place to start; revisit min_cases_for_alert and the per-metric warn/critical bands.

Dashboard

A Streamlit dashboard reads the same files the CLI does and renders them as a web UI. It is read-only: the dashboard never writes to runs/, never posts to Slack, never calls the LLM. To run an eval, use make eval. To browse runs, use the dashboard.

pip install -r requirements-dashboard.txt
make dashboard                # serves on http://localhost:8501

Pages:

  1. Overview — headline numbers for the latest run, the v1 -> latest diff (if both exist), dataset summary, drift summary.
  2. Runs — the timeline of every run record with sortable columns, a dropdown to drill into any single run, plus the per-slice table and confusion matrix from the run record.
  3. Diff — pick any two runs from the dropdowns, get the same four-axis diff the CLI produces, plus the regressed-cases table and the reasons.
  4. Drift — the rolling-N-run accuracy chart over time, with a table of the raw drift log entries.
  5. Dataset — the golden dataset with filters by category, difficulty, and edge tag, plus a per-case detail expander.

The dashboard is optional. The eval pipeline and the CI workflow do not require it. The tests/test_dashboard_smoke.py suite exercises the dashboard's data layer offline so the contract between the dashboard and the rest of the system is locked.

License

MIT. See LICENSE.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages