predictLearner for regr.h2o.gbm has returned a class character instead of a numeric!

[nagdevAmruthnath]

Description

By using h2o package for regression results in "returned a class instead on a numeric!" error. I have tried it with random forest, glm and gbm, it all results in the same error

Reproducible example

# load data
data(BostonHousing, package = "mlbench")  

# load package
library(mlr)  

# create a regression task
regr.task = makeRegrTask(id = "bh", data = BostonHousing, target = "medv", fixup.data="quiet")  

# make a learner
lrn = makeLearner("regr.h2o.gbm", predict.type = "response")  
]
# create resampling 
outer = makeResampleDesc("CV", iters = 3, stratify = FALSE)  

# if this is not set explicitly, it results in a error
wcol=NULL  

# running it
r = crossval(
  learner = lrn,
  task = regr.task,
  iters = 10
) 

Session info

Platform: x86_64-pc-linux-gnu (64-bit)  
Running under: Ubuntu 18.04.3 LTS  

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/openblas/libblas.so.3
LAPACK: /usr/lib/x86_64-linux-gnu/libopenblasp-r0.2.20.so

locale:  
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8       
 [4] LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                  LC_ADDRESS=C              
[10] LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:  
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:  
 [1] mlr_2.15.0        ParamHelpers_1.12 shiny_1.3.2       lubridate_1.7.4   dplyr_0.8.3      
 [6] caret_6.0-84      ggplot2_3.2.0     lattice_0.20-38   trend_1.1.1       h2o_3.22.1.1     
[11] RPostgreSQL_0.6-2 DBI_1.0.0        

loaded via a namespace (and not attached):  
 [1] Rcpp_1.0.2         shinyWidgets_0.4.8 class_7.3-15       assertthat_0.2.1   digest_0.6.20     
 [6] ipred_0.9-9        foreach_1.4.4      mime_0.7           R6_2.4.0           plyr_1.8.4        
[11] backports_1.1.4    stats4_3.6.1       esquisse_0.2.0     pillar_1.4.2       rlang_0.4.0       
[16] lazyeval_0.2.2     rstudioapi_0.10    data.table_1.12.2  miniUI_0.1.1.1     rpart_4.1-15      
[21] Matrix_1.2-17      checkmate_1.9.4    labeling_0.3       splines_3.6.1      gower_0.2.0       
[26] stringr_1.4.0      RCurl_1.95-4.12    munsell_0.5.0      compiler_3.6.1     httpuv_1.5.1      
[31] pkgconfig_2.0.2    BBmisc_1.11        htmltools_0.3.6    nnet_7.3-12        tidyselect_0.2.5  
[36] tibble_2.1.3       prodlim_2018.04.18 codetools_0.2-16   XML_3.98-1.19      viridisLite_0.3.0 
[41] crayon_1.3.4       withr_2.1.2        later_0.8.0        MASS_7.3-51.4      bitops_1.0-6      
[46] recipes_0.1.5      ModelMetrics_1.2.2 grid_3.6.1         xtable_1.8-4       nlme_3.1-140      
[51] jsonlite_1.6       gtable_0.3.0       magrittr_1.5       scales_1.0.0       stringi_1.4.3     
[56] reshape2_1.4.3     promises_1.0.1     ggthemes_4.2.0     parallelMap_1.4    timeDate_3043.102 
[61] generics_0.0.2     fastmatch_1.1-0    lava_1.6.5         RColorBrewer_1.1-2 iterators_1.0.10  
[66] tools_3.6.1        glue_1.3.1         purrr_0.3.2        parallel_3.6.1     survival_2.44-1.1 
[71] yaml_2.2.0         colorspace_1.4-1   extraDistr_1.8.10

Expected output

It should provided trained model results.

Actual output

Resampling: cross-validation
Measures:             mse       
  |==========================================================================================| 100%
  |==========================================================================================| 100%
  |==========================================================================================| 100%
  |==========================================================================================| 100%
Error in checkPredictLearnerOutput(.learner, .model, p) : 
  predictLearner for regr.h2o.gbm has returned a class character instead of a numeric!

PS: I looked in to the source and the result is as intended. This might be because of java object type conversion issues.

[pat-s]

Thanks for contributing @nagdevAmruthnath.

I can reproduce the issue.
I am not a h2o expert though.

PS: I looked in to the source and the result is as intended. This might be because of java object type conversion issues.

Can you elaborate a bit more? Is the "java conversion issue" on the mlr side?

[nagdevAmruthnath]

Thanks for contributing @nagdevAmruthnath.

I can reproduce the issue.
I am not a h2o expert though.

PS: I looked in to the source and the result is as intended. This might be because of java object type conversion issues.

Can you elaborate a bit more? Is the "java conversion issue" on the mlr side?

H2O runs on java run time (JDK) with API's to R. Example, a data frame in R is converted to java object. I think its probably some sort of type conversion. I am neither an expert. I am an end user of H2O.

[pat-s]

Ok. I do not think I can help here for now.

@berndbischl @mllg @larskotthoff
Does anyone of you have expertise in h2o and Java stuff?

[berndbischl]

i just debigged this.
the problem seems to be a bug in h2o as.data.frame?

this is the "p" object:

Class 'H2OFrame' <environment: 0x55f8fa0a4038>
 - attr(*, "op")= chr "predictions_a5f6_GBM_model_R_1570619765036_
10_on_data.frame_sid_ac60_27"
 - attr(*, "id")= chr "predictions_a5f6_GBM_model_R_1570619765036_
10_on_data.frame_sid_ac60_27"
 - attr(*, "eval")= logi FALSE
 - attr(*, "nrow")= int 51
 - attr(*, "ncol")= int 1
 - attr(*, "types")=List of 1
  ..$ : chr "real"
 - attr(*, "data")='data.frame':        10 obs. of  1 variable:
  ..$ predict: num  19.8 19.2 14.2 15.4 20.9 ...
NULL

this is the object after as.data.frame.H20Frame was called on it:

'data.frame':   51 obs. of  1 variable:
 $ predict: chr  "19.831134729691136" "19.170959664297577" "14.246
499434717286" "15.433545391795512" ...
NULL

of course we can sidestep the 2nd call and access the "data" attribute directly but i guess thats not clean and this should be reported upstream?
i also find it weird that our unit tests do not catch this? this should always fail not only in this example?

[nagdevAmruthnath]

@berndbischl I think it should be reported upstream. I am not sure if this is included in the unit test.

[pat-s]

I tracked it down. The pointer by @berndbischl was helpful but not the source of the issue. In fact there were two issues in this learner:

1. We aim to pass wcol as the weight column to the h2o::gbm function but we fail on initiating that variable the right way.

mlr/R/RLearner_regr_h2ogbm.R

Lines 58 to 74 in d30e54a

	trainLearner.regr.h2o.gbm = function(.learner, .task, .subset, .weights = NULL, ...) {
	# check if h2o connection already exists, otherwise start one
	conn.up = tryCatch(h2o::h2o.getConnection(), error = function(err) return(FALSE))
	if (!inherits(conn.up, "H2OConnection")) {
	h2o::h2o.init()
	options("h2o.use.data.table" = TRUE)
	}
	y = getTaskTargetNames(.task)
	x = getTaskFeatureNames(.task)
	d = getTaskData(.task, subset = .subset)
	if (!missing(.weights)) {
	wcol = ".mlr.weights"
	d[[wcol]] = .weights
	}
	h2of = h2o::as.h2o(d)
	model = h2o::h2o.gbm(y = y, x = x, training_frame = h2of, weights_column = wcol, ...)

2. While training the model during tests in

mlr/tests/testthat/helper_helpers.R

Line 67 in d30e54a

m = try(train(lrn, task, subset = inds))

all h2o.gbm models actually fail (due to (1)). Due to try() we simply continue to the predict step. There, bad things happen now with the prediction DF being in a wrong format.

If one solves (1), the issue of (2) goes away. I rewrote the h2o.gbm learners and tied them closely to the normal gbm learners in mlr. Now the train() call succeeds and also the predic.gbm call later on.

I also removed the try() call to see if we can live without it. Without it, tests would have failed way earlier and we would have seen the root of the issue way easier.

[berndbischl]

@pat-s thx a LOT for debugging this. can we pls make sure not to do these really bad try-catch calls anymore? that is IMHO the real issue here.

@mllg FYI