A hands-on QA project that evaluates a real, free, publicly available machine learning model the same way a professional test engineer would evaluate any AI system before it ships: measured precision/recall/F1 against a labeled dataset, plus adversarial/robustness testing and a fairness consistency probe -- not just a single, flattering accuracy number.
This was actually run. Every metric in reports/evaluation-report.md was
produced by executing real model inference on this machine (CPU, no GPU, no
API key, no paid service) against a 74-row hand-curated dataset. Nothing in
this repo's results section is illustrative or fabricated -- see
Results below and the raw JSON reports in reports/.
Most "AI testing" demos on GitHub either fake their numbers or only ever test the easy cases -- a handful of obviously-positive and obviously-negative sentences that any model gets right, followed by a proud "95% accuracy!" badge. That kind of evaluation tells you almost nothing about whether a model is safe to ship.
This project demonstrates the alternative: an evaluation designed to find where the model actually breaks. It measures:
- Accuracy / precision / recall / F1 against a labeled dataset, using
scikit-learn, broken down per category rather than reported as a single blended number. - Adversarial robustness -- does the same sentence, with typos, emoji, or stray punctuation, still get classified the same way?
- Sarcasm and mixed-sentiment handling -- the specific failure modes a binary sentiment classifier is most likely to have, tested deliberately rather than avoided.
- A fairness consistency probe -- framed honestly as a lightweight, directional sanity check, not a certified audit (see below).
distilbert-base-uncased-finetuned-sst-2-english(Hugging Face) is a small (~260MB), free, publicly available binary sentiment classifier. It requires no API key, no account, and no paid inference service -- it runs entirely on CPU, locally, in a few seconds per batch. That means anyone cloning this repo can reproduce the exact numbers below themselves.- Choosing a real, runnable model over a toy/mocked one matters: a test suite that only asserts trivial things about a stubbed model proves nothing about actual model behavior. This project's assertions and reports are only meaningful because real inference is happening.
dataset/test_cases.json contains 74 hand-authored rows across seven
categories, deliberately weighted toward the cases that make an evaluation
honest rather than flattering:
| Category | Rows | Why it's here |
|---|---|---|
baseline |
20 | Clear-cut sentiment -- the easy case, and the floor every model should clear. |
negation |
10 | Sentences like "not bad at all" that flip sentiment through negation -- tests whether the model actually parses structure or just counts positive/negative words. |
sarcasm |
8 | Sarcastic statements labeled by intended sentiment. Expected to score poorly -- that's the point: a baseline-only accuracy number would completely hide this failure mode. |
mixed_sentiment |
8 | Sentences with both positive and negative clauses, to see which one the model weighs more heavily. |
neutral_or_ambiguous |
8 | Genuinely ambiguous/factual statements. This binary model has no NEUTRAL class, so these rows test what it defaults to rather than scoring right/wrong. |
adversarial_typos |
10 | Each paired with a clean baseline sentence via typos, character swaps, dropped/duplicated letters, extra punctuation, and emoji injection -- to measure a real robustness delta, not just clean-input accuracy. |
fairness_probe |
10 (5 pairs) | Identical sentences except for a swapped name/pronoun/demographic term, to check prediction consistency. Framed explicitly as directional and exploratory -- see Fairness probe framing. |
If this dataset only contained the baseline category, the reported accuracy
would be 100% -- and it would be actively misleading about how the model
behaves on real-world text.
This is not a rigorous fairness audit. It is 5 hand-authored template sentence pairs checking whether swapping a name, pronoun, or demographic term changes the predicted label. A genuine fairness audit would need far more templates, systematic coverage of names/genders/ethnicities/age groups, statistical testing on confidence-score shifts (not just label flips), and review by people with relevant subject-matter expertise. Treat the result here as "no obvious red flag in this small sample," never as "certified fair."
At SEITech Solutions, I tested an AI-powered MISRA C/C++ static-analysis tool professionally -- measuring its precision and recall against graded test files containing known, mixed rule violations, and reporting where it over-flagged, under-flagged, or was inconsistent, rather than accepting a single headline accuracy number. This project applies that exact same discipline -- labeled ground truth, per-category breakdown, adversarial/edge- case probing, honest reporting of where the system under test is weak -- to a public, freely runnable ML model, so the methodology is fully visible and reproducible by anyone, without needing access to proprietary tooling or data.
git clone <this-repo>
cd qa-ml-model-evaluation-testing
python3 -m venv venv
source venv/bin/activate # Windows: venv\Scripts\activate
pip install -r requirements.txt # installs CPU-only torch + transformers;
# first run also downloads the ~260MB model
pytest tests/ -v -s # -s shows the per-category report tablesFirst run downloads the model from the Hugging Face Hub (cached under
~/.cache/huggingface afterward, so subsequent runs are fast and fully
offline). No API key, account, or payment is required at any point.
After running, see:
reports/evaluation-report.md-- the human-readable evaluation reportreports/accuracy_report.json,reports/robustness_report.json,reports/fairness_probe_report.json-- raw machine-readable results
Actually executed on: local CPU, Python 3.14, torch 2.13.0, transformers
5.13.1, scikit-learn 1.9.0. Full write-up: reports/evaluation-report.md.
| Measure | Result |
|---|---|
| Overall accuracy (66 binary-scorable rows) | 81.8% |
| Baseline-only accuracy | 100.0% |
| Negation accuracy | 80.0% |
| Mixed-sentiment accuracy | 87.5% |
| Sarcasm accuracy | 25.0% |
| Adversarial-typo accuracy | 70.0% |
| Robustness score (label stability under typo/emoji perturbation) | 70.0% |
| Fairness-probe consistency (5 templates, directional only) | 100.0% |
The headline finding: the model is essentially perfect on clear-cut sentiment (100% baseline) but gets 6 of 8 sarcastic statements backwards (25% accuracy) -- misreading surface-positive words like "wow", "fantastic", and "perfect" as genuine positivity even when the sentence is plainly sarcastic. It also flips label on 3 of 10 sentences after nothing more than typos and emoji, which matters for any product that has to handle unfiltered user-generated text. Full per-example breakdowns, the confusion matrix, and the robustness/fairness tables are in the evaluation report.
qa-ml-model-evaluation-testing/
├── dataset/
│ └── test_cases.json # 74 hand-labeled rows, 7 categories
├── src/
│ ├── model_runner.py # loads HF pipeline, predict(text) -> {label, score}
│ └── adversarial.py # deterministic typo/emoji/punctuation perturbation helpers
├── tests/
│ ├── conftest.py # shared fixtures: dataset load + single inference pass
│ ├── test_accuracy_metrics.py # accuracy/precision/recall/F1 + confusion matrix + per-category report
│ ├── test_robustness.py # clean vs. perturbed prediction stability
│ └── test_fairness_probe.py # directional fairness consistency probe
├── reports/
│ ├── evaluation-report.md # human-readable evaluation writeup (this run's real results)
│ ├── accuracy_report.json
│ ├── robustness_report.json
│ └── fairness_probe_report.json
├── .github/workflows/ml-tests.yml
├── requirements.txt
├── LICENSE
└── README.md
MIT -- see LICENSE.