handling columns called `class`

[simonpcouch]

stacks currently fails when a classification outcome name is class. The issue arises here:

stacks/R/add_candidates.R

Lines 282 to 285 in 4843385

	pred_class_idx <- grepl(pattern = ".pred_class", x = colnames(candidate_cols))
	candidate_cols <- candidate_cols[,!pred_class_idx] %>%
	setNames(., make.names(names(.)))

The logic here selects class probability predictions and the true outcome and gets rid of columns with .pred_class (i.e. the columns with hard class predictions), as the ensemble doesn't make use of hard class predictions in stacking models. {tune} (or some other dependency) labels hard class prediction columns as .pred_colname.

library(tidymodels)
library(workflowsets)
library(stacks)

# need devel modeldata for sim_classification
# devtools::install_github("tidymodels/modeldata")
library(modeldata)

x <- sim_classification(100)

spec_lasso <-
  logistic_reg(engine = 'glmnet', penalty = tune(), mixture = 1)

rec <- 
  recipe(class ~ ., x) %>% 
  step_normalize(all_numeric_predictors())

res <- tune_grid(
  spec_lasso,
  preprocessor = rec,
  resamples = vfold_cv(x, 2),
  grid = 2,
  control = control_stack_grid()
)

stacks() %>% 
  add_candidates(res) %>% 
  blend_predictions()
#> x Bootstrap01: preprocessor 1/1, model 1/1: Error in glmnet::glmnet(x = maybe_matrix(x)...
#> x Bootstrap02: preprocessor 1/1, model 1/1: Error in glmnet::glmnet(x = maybe_matrix(x)...
#> x Bootstrap03: preprocessor 1/1, model 1/1: Error in glmnet::glmnet(x = maybe_matrix(x)...
#> x Bootstrap04: preprocessor 1/1, model 1/1: Error in glmnet::glmnet(x = maybe_matrix(x)...
#> x Bootstrap05: preprocessor 1/1, model 1/1: Error in glmnet::glmnet(x = maybe_matrix(x)...
#> x Bootstrap06: preprocessor 1/1, model 1/1: Error in glmnet::glmnet(x = maybe_matrix(x)...
#> x Bootstrap07: preprocessor 1/1, model 1/1: Error in glmnet::glmnet(x = maybe_matrix(x)...
#> x Bootstrap08: preprocessor 1/1, model 1/1: Error in glmnet::glmnet(x = maybe_matrix(x)...
#> x Bootstrap09: preprocessor 1/1, model 1/1: Error in glmnet::glmnet(x = maybe_matrix(x)...
#> x Bootstrap10: preprocessor 1/1, model 1/1: Error in glmnet::glmnet(x = maybe_matrix(x)...
#> x Bootstrap11: preprocessor 1/1, model 1/1: Error in glmnet::glmnet(x = maybe_matrix(x)...
#> x Bootstrap12: preprocessor 1/1, model 1/1: Error in glmnet::glmnet(x = maybe_matrix(x)...
#> x Bootstrap13: preprocessor 1/1, model 1/1: Error in glmnet::glmnet(x = maybe_matrix(x)...
#> x Bootstrap14: preprocessor 1/1, model 1/1: Error in glmnet::glmnet(x = maybe_matrix(x)...
#> x Bootstrap15: preprocessor 1/1, model 1/1: Error in glmnet::glmnet(x = maybe_matrix(x)...
#> x Bootstrap16: preprocessor 1/1, model 1/1: Error in glmnet::glmnet(x = maybe_matrix(x)...
#> x Bootstrap17: preprocessor 1/1, model 1/1: Error in glmnet::glmnet(x = maybe_matrix(x)...
#> x Bootstrap18: preprocessor 1/1, model 1/1: Error in glmnet::glmnet(x = maybe_matrix(x)...
#> x Bootstrap19: preprocessor 1/1, model 1/1: Error in glmnet::glmnet(x = maybe_matrix(x)...
#> x Bootstrap20: preprocessor 1/1, model 1/1: Error in glmnet::glmnet(x = maybe_matrix(x)...
#> x Bootstrap21: preprocessor 1/1, model 1/1: Error in glmnet::glmnet(x = maybe_matrix(x)...
#> x Bootstrap22: preprocessor 1/1, model 1/1: Error in glmnet::glmnet(x = maybe_matrix(x)...
#> x Bootstrap23: preprocessor 1/1, model 1/1: Error in glmnet::glmnet(x = maybe_matrix(x)...
#> x Bootstrap24: preprocessor 1/1, model 1/1: Error in glmnet::glmnet(x = maybe_matrix(x)...
#> x Bootstrap25: preprocessor 1/1, model 1/1: Error in glmnet::glmnet(x = maybe_matrix(x)...
#> Warning: All models failed. See the `.notes` column.
#> Error in `estimate_tune_results()` at tune/R/select_best.R:75:2:
#> ! All of the models failed. See the .notes column.

^{Created on 2022-04-28 by the reprex package (v2.0.1)}

stacks ought to handle that column selecting more elegantly.

Reported in #120.

[simonpcouch]

Another edge case I'd anticipate we need to handle—ensure this works when a level of an outcome is "class". Wheeee

[mattwarkentin]

I was just about to report this. It seems like you can't have the word "class" in your outcome levels at all. I changed the name of the variable class to y and it still failed. Renaming the variable AND the levels got things back to working. Just wanted to let you know.

[mattwarkentin]

Side question, what does it mean when a model contributes a non-integer number of candidate members? Example:

# A data stack with 6 model definitions and 27.5 candidate members:
#   std_LASSO: 3 model configurations
#   std_SVM: 5 model configurations
#   std_LDA: 5 model configurations
#   std_KNN: 5 model configurations
#   std_XGBoost: 4.5 model configurations
#   std_RandFor: 5 model configurations
# Outcome: y (factor)

[simonpcouch]

Please file separate issues for asides and provide a reprex along with them! Much appreciated. :)

[github-actions]

This issue has been automatically locked. If you believe you have found a related problem, please file a new issue (with a reprex: https://reprex.tidyverse.org) and link to this issue.