Performance tests for Fairlearn

This repository exclusively contains performance tests for the Fairlearn repository.

Running tests locally

Setup:

# clone Fairlearn repo and Fairlearn-Performance repo
git clone https://github.com/fairlearn/fairlearn.git
git clone https://github.com/fairlearn/fairlearn-performance.git
# install Fairlearn-Performance dependencies
pip install -r fairlearn-performance/requirements.txt
# install latest Fairlearn to ensure compatibility
pip install fairlearn/.

# to show all tests
python -m pytest .\fairlearn-performance\perf\ --maxfail=1 -s --collect-only
# to run a single test from the full list
python -m pytest .\fairlearn-performance\perf\ --maxfail=1 -s -k test_perf[[dataset:adult_uci,estimator:DecisionTreeClassifier[],mitigator:GridSearch,disparity_metric:TruePositiveRateParity[]]]