support to override some parameters in Dataset

[guolinke]

fix #1853

[guolinke]

ping @Laurae2
@Laurae2 if you are not available recently, we can merge the modification in python first.

[mayer79]

@guolinke: Great. Below a proposal for R modifying the "update_params" function in R-Package/R/lgb.Dataset.R [as I am not familiar how to commit to a pull request].

The current implementation for Python (and R) has two minor issues. I think we can live with them for the moment:

1. If we e.g. use "min_data = 10" instead of "min_data_in_leaf = 10", the warning will be raised unnecessarily.

2. If we e.g. start with "min_data = 1", then set "min_data = 10" we correctly get a warning. Then, if we reset the parameters back to the original "min_data = 1", we will again see a warning, even if the parameters are now in line with those used for constructing the data set.

   # Update parameters
   update_params = function(params) {
     
     # Check if new parameter list is different from old list wrt problematic parameters.
     if (!lgb.is.null.handle(private$handle)) {
       key_list <- c("min_data", "min_data_in_leaf", "min_data_per_leaf", "min_child_samples",
                     "bin_construct_sample_cnt", "subsample_for_bin",
                     "max_bin", 
                     "min_data_in_bin",
                     "enable_bundle", "is_enable_bundle", "bundle",
                     "max_conflict_rate",
                     "is_enable_sparse", "sparse", "is_sparse", "enable_sparse",
                     "sparse_threshold",
                     "use_missing",
                     "zero_as_missing",
                     "categorical_feature", "cat_feature", "categorical_column", "cat_column",
                     "monotone_constraints", "monotone_constraint", "mc")
       
       for (key in names(params)) {
         if (key %in% key_list && (!(key %in% names(private$params)) || private$params[[key]] != params[[key]])) {
           warning("Cannot update parameter ", sQuote(key), 
                   " after construct Dataset. Please re-declare a new Dataset object with new parameters.")
         }
       }
     } 
     
     # Overwrite parameters, even if inconsistent with Dataset construction.
     if (is.na(private$params) || !length(private$params)) {
       private$params <- params
     } else {
       private$params <- modifyList(private$params, params)
     } 
     
     return(invisible(self))
     
   },
   

[guolinke]

@mayer79
the second issue will not happen, as the parameter will be reserved after using a new booster:
https://github.com/Microsoft/LightGBM/pull/1876/files#diff-732a5a5220860efcac575e9e956bbaeaL1076

The alias indeed is a problem, but support it is not trival.
I think we can live with it now.

[mayer79]

@guolinke: In Python, this is indeed implemented. In R, I could not spot a reverse update in train.R though.

[guolinke]

todo: the same functionality in R.

[Laurae2]

@guolinke If I understand correctly, we check the list of parameters before training so if another parameter in that list shows up, then we throw a warning? (would need to check if dataset is constructed beforehand or not)

By design in R should we enforce dataset parameters in lgb.Dataset only or should we allow them the first time in lgb.train?

[guolinke]

@Laurae2 sorry for late response.
I think the warning condition is: user try to override existing parameters, after the handle of dataset is constructed.
As the handle of dataset is lazily constructed, it is not enough to just enforce this before in lgb.Dataset or lgb.train.

[guolinke]

@Laurae2
Actually, I think this should be done inside cpp. as there are alias and default values (which is not show in python/R).
I will move these logics into cpp side when I have time.

[guolinke]

@mayer79 @Laurae2
Please check the new implementation.
Now, the mc and feature_contrib could be updated

[guolinke]

ping @mayer79 @Laurae2

[mayer79]

I tested the commit

commit a3595d00a0ae2e0c7e56b24135d589a2301c4e61 (HEAD, origin/re-construct-dataset)
Author: Guolin Ke <guolin.ke@outlook.com>
Date:   Fri Dec 28 15:48:09 2018 +0800

    Update basic.py

in R with the following findings:

The first time when I run lightgbm in a session, I get a warning:

Warning message:
In charToRaw(as.character(x)) :
  argument should be a character vector of length 1
all but the first element will be ignored

This happens when update_params(params) is called for the second time.

debug: data$update_params(params)
Browse[2]> 
[LightGBM] [Warning] Unknown parameter 0.03
debug: data$.__enclos_env__$private$set_predictor(predictor)

I have never seen this warning with older versions of the code, so it is probably due to the changes.

Then, if I change a problematic parameter (min_data_in_leaf from 10 to 20) and run lgb.train again, there is no warning at all. So it is not yet fully compatible with the R wrapper.

This is the reproducible code used:

library(tidyverse)
library(caret)
library(lightgbm)

# Generate function with one input x
set.seed(20)
x <- seq(-0.2, 1, by = 0.001)
y <- x^2 + rnorm(length(x), sd = 0.1)

dat <- data.frame(x, y)

# train/test split
ind <- caret::createDataPartition(y, p = 0.8, list = FALSE) %>% c
train <- dat[ind, ] %>% data.matrix
test <- dat[-ind, ] %>% data.matrix

# LGB
lgb_data <- lgb.Dataset(train[, "x", drop = FALSE], label = train[, "y"])

# Some reasonable parameters (would need to be optimized)
params <- list(learning_rate = 0.03,
               num_leaves = 31,
               min_data_in_leaf = 10,
               bagging_fraction = 0.7,
               bagging_freq = 4,
               nthread = 4,
               max_bin = 255,
               metric = "rmse",
               objective = "regression")

# Fit for first time gives warning
fit <- lgb.train(params = params, 
                 data = lgb_data, 
                 nrounds = 2)

# Now change tricky param
params$min_data_in_leaf <- 20

lgb.train(params = params, 
          data = lgb_data, 
          nrounds = 100)

[guolinke]

Thanks @mayer79, there is a bug in R. just fix it.

[mayer79]

I am afraid that there is still a small problem with that call:

data$update_params(params)
[LightGBM] [Warning] Unknown parameter 6c
Warning message:
In charToRaw(as.character(x)) :
  argument should be a character vector of length 1
all but the first element will be ignored
Browse[2]> lgb.params2str(params)
  [1] 6c 65 61 72 6e 69 6e 67 5f 72 61 74 65 3d 30 2e 30 33 20 6e 75 6d 5f 6c 65 61 76 65 73 3d 33 31 20 6d 69 6e 5f 64
 [39] 61 74 61 5f 69 6e 5f 6c 65 61 66 3d 31 30 20 62 61 67 67 69 6e 67 5f 66 72 61 63 74 69 6f 6e 3d 30 2e 37 20 62 61
 [77] 67 67 69 6e 67 5f 66 72 65 71 3d 34 20 6e 74 68 72 65 61 64 3d 34 20 6d 61 78 5f 62 69 6e 3d 32 35 35 20 6d 65 74
[115] 72 69 63 3d 72 6d 73 65 20 6f 62 6a 65 63 74 69 76 65 3d 72 65 67 72 65 73 73 69 6f 6e 20 76 65 72 62 6f 73 65 3d
[153] 31 00

Should it be

lgb.call("LGBM_DatasetUpdateParam_R", ret = NULL, private$handle, 
    lgb.params2str(params))

instead of

lgb.call("LGBM_DatasetUpdateParam_R", ret = NULL, private$handle, 
    lgb.c_str(lgb.params2str(params)))

within data$update_params?

[guolinke]

@mayer79 yeah, you are right! thanks for point it out

[mayer79]

Oops, now fatal error & crash ;)

[guolinke]

Any logs?

[mayer79]

I am afraid no - RGui crashes without log. So far I was unable to collect a dump file in R Windows.

[guolinke]

@mayer79
sorry, I find another bug.
Very sorry, I am very busy and cannot verify these codes carefully...

[mayer79]

I did not spot that one - crap! I will test after work. No need to apologize - your work is incredibly useful.

[mayer79]

Now it works: After changing "max_bin", the following message pops up

[LightGBM] [Warning] Cannot change max_bin after constructed Dataset handle.

If I change the parameter back to its original value, the warning vanishes.

So the behaviour is as desired, thanks!

Is the parameter "min_data" aka "min_data_in_leaf", "min_data_per_leaf", "min_child_samples" not in the list of problematic factors? I remember that the python prototype mentioned it but now it can be updated without warning. Would be fantastic as this is one of the most important parameters to tune.

[guolinke]

@mayer79
The min_data is used to filter some features that cannot split. For example, when min_data=20 and one feature with 10 0s and 30 1s, this feature cannot be splitted. This is for the efficiency purpose.
Therefore, Updating 'min_data` in dataset will not affect the result, as these filter features will not be chosen by trees as well.