[Question] Cross validation and Ensembling

[FelixNeutatz]

Dear all,

I am wondering how greedy ensembling is implemented for cross-validation. I couldn't really find it in the code. Can anybody give me a hint?

My idea of how it could be implemented:

weights = np.zeros(len(models))
ensemble_sel = EnsembleSelection(ensemble_size=50,
                                      task_type=MULTICLASS_CLASSIFICATION,
                                      random_state=0,
                                      metric=ba)

for k in range(cv_folds):
      validation_indices = get_validation_ids(k)
      ensemble_sel.fit(model_val_predictions[k][validation_indices], y_test[validation_indices], identifiers=None)
      weights += ensemble_sel.weights_
ensemble_sel.weights_ = weights_ / cv_folds

Is this roughly how it works?

Best regards,
Felix

[eddiebergman]

Hi @FelixNeutatz,

I spent a while digging through and it seems like we basically consider the cv predictions as one big long array vertically stacked, metric'd with the corresponding targets vertically stacked:

# Not real code but just to use your frame of reference
# Happens in TrainEvaluator
cv_predictions = [[1, 1, 1], [1, 0, 1], [0, 0, 0]]
cv_targets = [[1, 1, 1], [0, 1, 1], [0, 0, 0]] 
stacked_predictions = np.concatenate(cv_predictions)
stacked_targets = np.concatenate(cv_targets)

# Save these to disk, notice how we don't care if it's CV now
save(id="...", stacked_predictions, stacked_targets)

#
# ...
#

# Somewhere else while ensemble building, we can feed these to ensemble selection
predictions, targets = load(id="...")

---

I'd like to point you to one place in the code where this happens but it's convoluted around several other points. Here's my investigation stacktrace if curious:

• The EnsembleBuilder is responsible for creating the EnsembleSelection and fitting it. After it prunes some memory and selects candidates we consider for ensemble selection, we create an EnsembleSelection and call fit_ensemble here
  

  auto-sklearn/autosklearn/ensemble_building/builder.py

  Lines 566 to 577 in 013d7ee

  	ensemble = self.fit_ensemble(
  	candidates=candidates,
  	targets=targets,
  	runs=runs,
  	ensemble_class=self.ensemble_class,
  	ensemble_kwargs=self.ensemble_kwargs,
  	X_data=ensemble_X_data,
  	task=self.task_type,
  	metrics=self.metrics,
  	precision=self.precision,
  	random_state=self.random_state,
  	)

• Here is the fit method you have in your sample code, called inside the fit_ensemble function above. It loads in predictions for the candidate runs but never seems to care whether it's cross validation or not, must be agnostic to it somehow.
  

  auto-sklearn/autosklearn/ensemble_building/builder.py

  Lines 932 to 942 in 013d7ee

  	predictions_train = [
  	run.predictions("ensemble", precision=precision) for run in candidates
  	]
  	ensemble.fit(
  	base_models_predictions=predictions_train,
  	X_data=X_data,
  	true_targets=targets,
  	model_identifiers=[run.id for run in candidates],
  	runs=runs,
  	)

• Following the trace of where these predictions get loaded from:

  • auto-sklearn/autosklearn/ensemble_building/run.py

    Lines 128 to 133 in 013d7ee

    	path = self.pred_path(kind)
    	with path.open("rb") as f:
    	# TODO: We should probably remove this requirement. I'm not sure why model
    	# predictions are being saved as pickled
    	predictions = np.load(f, allow_pickle=True)

  • auto-sklearn/autosklearn/ensemble_building/run.py

    Lines 77 to 80 in 013d7ee

    	def pred_path(self, kind: str = "ensemble") -> Path:
    	"""Get the path to certain predictions"""
    	fname = f"predictions_{kind}_{self.seed}_{self.num_run}_{self.budget}.npy"
    	return self.dir / fname

  • Sigh, we've lost the follow-able trace, need to find where predictions are saved
  • It's this file TrainEvaluator
  • Calls this finish_up and file_output at some point which dumps files.
  • Goes over here now save_numrun_to_dir. It basically saves cv_models and regular model predictions to the same place, backing up the previous assumption
• Back to TrainEvaluator and where it generates this one prediction file per cv evaluation of a pipeline.
• Won't go into detail here but this is the cv branch of the TrainEvaluator where it generates these predictions
• At this point we concat it together as mentioned above

[FelixNeutatz]

Thank you so much! This is really helpful. Maybe this should go into the documentation :)

[eddiebergman]

I'll keep it open as something to add into the docs then!

[BradKML]

TrainEvaluator looks like something that belongs in this documentation, or are they different? https://automl.github.io/auto-sklearn/master/examples/40_advanced/example_resampling.html#cross-validation

[eddiebergman]

The TrainEvaluator is not something that's exposed to the user so I don't think it would go in there personally, I'm not sure how it would look belonging in there.