Tests / Checks for flipped factors

[sebffischer]

I can imagine that some learner's don't handle factors with flipped levels perfectly.
A proposal would be to add an autotest that checks whether predictions are equivalent irrespective of the underlying factor representation.
Then, there could be a learner property that indicates whether the learner can deal with flipped factor levels.
In the $predict_newdata() and the $predict() method we could then (depending on whether this property is present) throw an error.

[sebffischer]

Decision: We add it as an autotest (whether a learner can handle it) and then check where we have to fix stuff

[berndbischl]

Please either show a reprex where this breaks the package or close during the workshop

[sebffischer]

The reprex below shows that:

• we don't fix the ordering of factor levels
• some learners cannot handle this (maybe because they convert factors to integers via as.integer (?))

library(mlr3)
library(mlr3learners)
library(mlr3extralearners)

learner = lrn("classif.ksvm")

x = sample(c("a", "b"), 100, replace = TRUE)
y = x
y[sample(1:100, 10)] = sample(c("a", "b"), 10, replace = TRUE)

d = data.table(
  x = factor(x, levels = c("a", "b")),
  y = factor(y)
)

dflip = data.table(
  x = factor(x, labels = c("b", "a"), levels = c("b", "a")),
  y = factor(y)
)

task = as_task_classif(d, target = "y")
task_flip = as_task_classif(dflip, target = "y")

learner$train(task)

p1 = learner$predict(task)
p2 = learner$predict(task_flip)

stopifnot(all(p1$response != p2$response))

^{Created on 2024-08-16 with reprex v2.0.2}