Skip to content

feat(ml): ensemble classifiers from cv - #99

Draft
junkha wants to merge 6 commits into
Bayer-Group:mainfrom
junkha:90-ensemble-classifiers-from-cv
Draft

junkha wants to merge 6 commits into
Bayer-Group:mainfrom
junkha:90-ensemble-classifiers-from-cv

Conversation

@junkha

@junkha junkha commented Oct 5, 2026

Copy link
Copy Markdown

Summary

This PR introduces ensemble classifiers that combine fitted estimators produced by mother_cv for prediction and uncertainty quantification.

Changes

  • Introduce CVEnsembleClassifierMother (see Introduce ensemble models by combining trained models from MotherCV #90)

  • Combine fitted estimators from cross-validation runs into an ensemble classifier

  • Provide default option for aggregation of member predictions: averaging without weighting

  • Provide options for estimation of knowledge uncertainty:

    1. variance: computes variance of ensemble predictions over classes and members to represent ensemble uncertainty
    2. disagreement: computes disagreement score using KL-divergence to represent ensemble uncertainty
    3. member_average: computes ensemble uncertainty by averaging individual member uncertainties
  • Fix estimator handling in mother_cv when tuning is disabled (see # mother_cv returned identical member estimators without nested tuning #92)

  • Add example notebook demonstrating usage

  • Add unit tests

Motivation

The goal is to enable ensemble predictions from models produced during cross-validation while keeping the existing workflow for model training and evaluation.

Testing

  • Added automated tests covering ensemble classifier behavior
  • Verified existing test suite continues to pass
  • Validated example notebook

Notes

This is currently a draft PR.
I may still make minor adjustments and rebase onto the latest upstream main before requesting formal review.

Open To do's

  • review unit tests covering 'mother_cv' usage for ranking with CatBoost - 2 tests failing right now

Follow-up tasks

  • Provide more options for aggregation: weighted average, majority vote, etc.
  • Provide more options for ensemble uncertainty estimation: BALD/ mutual information, etc.
  • Add calibration methods for probability and uncertainty calibration of individual ensemble members based on their cv validation fold from fitting with mother_cv

@junkha
junkha force-pushed the 90-ensemble-classifiers-from-cv branch from 616d30e to 704aadc Compare October 6, 2026 07:37
@junkha
junkha force-pushed the 90-ensemble-classifiers-from-cv branch from 57b07bf to 0ec83c9 Compare October 8, 2026 11:31

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant