TypeError: 'NoneType' object is not iterable

[pseudotensor]

fail.zip

Code was compiled for GPU. Uses hash: d169ac8

Issue has been there for many months, but we noticed it happening for non-trivial data as well so now posting issue with example.

[guolinke]

The root cause is the nan in evaluation score, and cause the wrong comparison result in early stopping.

[pseudotensor]

So you don't have concerns about the data itself or the parameters? And the above fix should be good for these cases we see (many, especially for objectives other than plain regression) this score error -> NoneType issue?

[guolinke]

In gamma regression, the label should greater zero;
https://github.com/Microsoft/LightGBM/blob/52dbc5722fe0b80d224ee33ba5b064b80966e6c6/src/metric/regression_metric.hpp#L255
But in your code, there are many labels with zeros.
I think we should throw the error in these cases.

[pseudotensor]

yes a direct error would be good

[pseudotensor]

I think main issue here is that lightgbm thinks no trees are better than any amount of trees, and then the score becomes bad.

This is related to an issue before, when lightgbm would fail to generate at least an average (for regression) or most-common (for classification) target as the base model.

Yes, if negative labels (or <=0 for gamma), then agreed an error.

[pseudotensor]

I'll try to find another case that isn't with a less common objective.

[pseudotensor]

fail5.zip

Here's another case that fails in the same way. Yes, features are funny, but target is normal. Shouldn't fail. Should at least be able to recover average (for regression) or most frequent (for classification) model.

[guolinke]

@pseudotensor Actually, LightGBM have dummy trees for these cases.
I edit your pasted code for test that.

import _pickle as pickle
import pandas as pd
import numpy as np
import lightgbm as lgb
model = lgb.LGBMRegressor(bagging_seed=3137599, boosting_type='gbdt', class_weight=None,
                             colsample_bytree=0.8, 
                             feature_fraction_seed=3137598, gpu_device_id=1, gpu_platform_id=0,
                             importance_type='gain', learning_rate=0.05, max_bin=255,
                             max_delta_step=0, max_depth=-1, min_child_samples=1,
                             min_child_weight=1, min_data_in_bin=1, min_split_gain=0.0,
                             n_estimators=600, n_jobs=4, num_class=1, num_leaves=64,
                             objective='regression', random_state=None, reg_alpha=0.0,
                             reg_lambda=1.0, scale_pos_weight=1, silent=True, subsample=1.0,
                             subsample_for_bin=200000, subsample_freq=1, verbose=2)
X, y, sample_weight, init_score, eval_set, valid_X_features, eval_sample_weight, init_score, eval_metric, early_stopping_rounds, X_features, verbose_fit = pickle.load(open("data.pkl", "rb"))
y = y/y.max()
valid_y = eval_set[0][1]
# below uncommented has no issues
#valid_y = valid_y/valid_y.max()
eval_set[0] = (eval_set[0][0], valid_y)
model.fit(X, y)
p = model.predict(eval_set[0][0])

and the values of p is

array([0.33332364, 0.33332364, 0.33332364, ..., 0.33332364, 0.33332364,
       0.33332364])

The fail reason is the same as above case, due to the l2 loss is inf (too large range of test label).

[pseudotensor]

Ok, I think as long as the error is improved, this can be closed.

[pseudotensor]

Here is a corner case that could be fixed too:

fail6.zip

Basically all sample weights for the eval set are 0. But even if the eval set can't be used for some reason, should at least be able to predict the average (dummy trees) model.

[guolinke]

@pseudotensor
It actually has the dummy trees. tested by your code

model.fit(X, y, sample_weight=sample_weight, init_score=init_score, eval_set=eval_set, eval_init_score=init_score, eval_metric=eval_metric, eval_sample_weight=eval_sample_weight)
model.predict(X)
model.predict(eval_set[0][0])

As well as the early_stopping_rounds is removed, the model can provide the average prediction.

[pseudotensor]

Understood, but even if there is a bad eval_set or early stopping, the worse case should be dummy trees always. It shouldn't have failed even with early stopping turned on.

[guolinke]

@pseudotensor yeah, the fixes are in #1973
and your cases are added as test cases 😄