machine-learning-model-selection-lab

How model-selection and validation choices create apparently excellent ML performance that disappears under the matching evaluation design.

Small, justified subsets of learners are fit on documented synthetic DGPs. The invalid workflow's number is kept next to the scientifically matched number. Those draws are method checks, not empirical findings.

Dr. Pavanam Thomas · pavanamthomas · thomaspavanam@gmail.com

A high score is not evidence that the estimand was the intended one. Perfect ROC-AUC can coexist with worse Brier score and log loss.

What quantity is being estimated

Read in this order:

1. CASE_STUDY_MODEL_SELECTION_FAILURE.md — three error locations (preprocessor, splitter, inner best_score_) that manufacture a headline.
2. docs/failures_and_corrections.md — the ten designed failures and the tests that keep them visible.
3. docs/leakage_taxonomy.md and docs/validation_design.md — named leaks and named estimands.
4. src/mslab/experiments/ — each module returns naive_score, correct_score, diagnostics, and a what_failed paragraph.
5. tests/ — property checks (disjoint splits, future sentinels, group isolation, train-only scalers, nested inner/outer indices, informative vs noise, ECE, seed 2026, majority recall 0).
6. src/mslab/models/catalogue.py — a catalogue exists; JUSTIFIED_SUBSETS forbids running every model on every DGP.

Reproduce from a clean clone:

python -m pip install -e .
python -m pytest
python scripts/run_all.py

Python 3.11 or newer. Printed numbers come from that script. They are not hard-coded in this README.

Estimands and designs

• Estimand first. Every splitter docstring in validation/designs.py names the risk it estimates. GroupKFold is not “more rigorous KFold”; it is a different sampling unit.
• Leakage is a line of code. The invalid preprocessor is fit_preprocessor_on_full_frame. The valid one is fit_preprocessor_on_train. Tests compare scaler means.
• Nested CV does not report the inner score. nested_cv_evaluate returns the outer mean. Inner best_score_ is a diagnostic.
• Metrics are functionals. Accuracy, ROC-AUC, PR-AUC, Brier, and ECE answer different questions. Majority accuracy on prevalence 5% is a closed-form trap: expected accuracy = 0.95, recall = 0.
• Permutation importance is not causal. I use it, and I say so in docs/model_interpretation_limits.md. SHAP is not a dependency.
• Justified subsets. Logistic for leakage and snooping; a tree grid for search optimism; a shallow tree for covariate shift (local leaves, source support); a forest for impurity ranks and overconfident probabilities; a dummy next to a forest when there is no signal.
• Synthetic DGPs with ground truth. Informative vs noise, correlated proxy, grouped intercepts, rare events, time plus a future sentinel, covariate shift. No observational extract.

Designed counterexamples

These are the design, not accidents:

1. Supervised selection fitted on all rows inflates holdout AUC.
2. Shuffled KFold with entity codes overstates risk for a new entity.
3. A post-outcome sentinel plus a shuffled split overstates forward risk.
4. GridSearchCV.best_score_ sits above nested outer accuracy.
5. Majority accuracy is high; rare-class recall is zero.
6. Full-sample correlation screening scored on the same rows overstates AUC.
7. Impurity ranks a correlated proxy that has structural coefficient 0.
8. Forest ROC-AUC looks strong while ECE does not.
9. Source holdout is not target risk under covariate shift.
10. Forest training accuracy on labels independent of x is memorisation.

Locked by docs/failures_and_corrections.md and the tests cited there.

Property checks

Tests check properties, not merely that a function returns an object:

• Holdout indices partition {0,…,n−1} with empty intersection.
• TimeSeriesSplit trains only on earlier times. The future sentinel is in the raw DGP and is dropped on the correct path.
• GroupKFold never shares an entity; shuffled KFold does on the grouped DGP.
• Leaky scaler means match a full-frame fit and differ from the training-fold scaler.
• Nested CV audits: outer test ∩ inner train = ∅.
• On noise_features_dgp, mean permutation importance of informative columns exceeds noise.
• True Bernoulli probabilities have small ECE; an overconfident 0.01/0.99 map has larger ECE.
• Seed 2026 reproduces experiment scores.
• Majority classifier: accuracy ≥ 0.90, recall = 0 on the 5% DGP.

Reproducibility

python -m pip install -e .
python -m pytest
python scripts/run_all.py

scripts/run_all.py writes figures under outputs/figures/ and a numeric summary to outputs/tables/run_summary.csv. Those files are regenerable. The source of truth is the code plus the tests.

CI installs the package, runs pytest, and runs scripts/run_all.py with MPLBACKEND=Agg.

Closed-form identity that does not depend on a lucky draw: a majority classifier on prevalence 5% has expected accuracy 0.95 and recall 0. Simulation quantities are printed by the script.

Known limitations

• The DGPs are stylised. They are tools for checking procedures, not models of a clinic, a credit book, or a platform.
• A score gap on a designed DGP is not a meta-analysis of published papers.
• Nested CV estimates the risk of a selection procedure under the splitter's sampling model. It is not unbiased in finite samples in a theorem proved here.
• Permutation importance, impurity, and coefficients are not causal effects. Correlation between x1 and x2 is enough to make unique attribution unidentified.
• No result in this repository is a claim about a deployed system, a client, or a production latency.
• Related causal identification work lives in a separate laboratory.

Remaining selection-procedure bounds

Purged/embargoed splits beyond TimeSeriesSplit(gap=...) are not implemented. Importance-weighted target-risk estimation under covariate shift is not implemented. Nested CV estimates the risk of a selection procedure under the splitter's sampling model; it is not proved unbiased here. See ROADMAP.md.

Repository structure

machine-learning-model-selection-lab/
├── README.md
├── CASE_STUDY_MODEL_SELECTION_FAILURE.md
├── LICENSE
├── CITATION.cff
├── pyproject.toml
├── docs/
├── src/mslab/
│   ├── dgp/
│   ├── preprocess/
│   ├── validation/
│   ├── selection/
│   ├── metrics/
│   ├── models/
│   ├── importance/
│   └── experiments/
├── scripts/run_all.py
├── tests/
├── outputs/
└── .github/workflows/ci.yml

Data

There is no observational dataset. Every draw is generated in code. Provenance and regeneration: docs/data_policy.md.

Related repositories

• statistical-reasoning-validation — probability identities, interval coverage, classification denominators.
• econometrics-causal-inference-lab — estimands when the question is causal.
• ai-response-evaluation-benchmarks — structured review of quantitative arguments.

Citation

See CITATION.cff. Licence: MIT, Copyright 2026 Dr. Pavanam Thomas.