testing-ml

Examples on how to test machine learning code. We'll test a numpy implementation of DecisionTree and RandomForest, covering some standard software tests, model tests, and model evaluation.

Inspired by @jeremyjordan's Effective Testing for Machine Learning Systems; follow-up article on 2020-09-06 @ eugeneyan.com.

Quick Start

# Clone and setup environment
git clone git@github.com:eugeneyan/testing-ml.git
cd testing-ml
make setup

# Run test suite
make check

Standard software tests

• Unit tests & Code coverage
• Linting
• Type checking

Model tests

• Pre-train tests

  • Check on test dataset shape
  • Checking output shape and range
  • Check if if model can overfit perfectly
  • Check if additional DecisionTree depth increases training accuracy
  • Check if additional RandomForest trees increases validation accuracy

• Post-train tests

  • Check invariance (i.e., change in input doesn't affect output)
  • Check directional expectation (i.e., output changes in an expected manner)
  • Check minimum functionality (e.g., with null values)
  • Check Random Forest outperforms Decision Trees

Model evaluation

• Evaluation on train-test split
  • Decision Tree Accuracy and AUC ROC
  • Random Forest Accuracy and AUC ROC