Python - Different handling of similar looking data in LGBMClassifier

[vivekk0903]

Related to the question: https://stackoverflow.com/questions/44558435/scikit-learn-predicting-a-single-observation.

The following lines are handled differently by the python LGBMClassifier:

clf.predict_proba(X[1])             # Throws an error about data not being 2-d.
clf.predict_proba(list(X[1]))      # Takes the data as single row, and produces results.

Environment info

Linux-3.16.0-77-generic-x86_64-with-Ubuntu-14.04-trusty
('Python', '2.7.6 (default, Oct 26 2016, 20:30:19) \n[GCC 4.8.4]')
('NumPy', '1.13.0')
('SciPy', '0.19.0')
('Scikit-Learn', '0.18.1')

Reproducible example

from sklearn.datasets import load_iris
data = load_iris()
X = data.data
y = data.target

from lightgbm import LGBMClassifier

clf = LGBMClassifier()
clf.fit(X,y)

# Case 1
clf.predict_proba(X[1])

#Case 2
clf.predict_proba(list(X[1]))

Explanation

Case 1 raises an error:

raise ValueError('Input numpy.ndarray must be 2 dimensional')

Case 2 though looks almost same (a list instead of numpy array) but interpreted as a row of data with its elements considered as the features of that row and the output is correct.

I wanted to know if this is the intended behaviour or just a case missed by the prediciton code. So please feel free to close this if not suitable.

[Laurae2]

In case 1 you have a 1d data, when in case 2 it is 2d. I think this is an expected behavior to throw an error on 1st case but not on 2nd.

@wxchan ^ ?

[vivekk0903]

@Laurae2 No, case 2 is not 2-d. When the data is passed to LGBMClassifier, its is a single list of 4 elements.
[4.9000000000000004, 3.0, 1.3999999999999999, 0.20000000000000001]

[Laurae2]

Your list is 2d minimum because a list in Python can contain an arbitrary number of elements which contains themselves an arbitrary number of elements. Therefore it is the expected behavior. (Talking about in memory representation, not the printed representation which has nothing to do about in memory).

[vivekk0903]

Yes I agree. But in a case where the training data has only 1 feature, how will this work? It will lead to wrong results.

Lets say, I am training a single feature data now.

X_new = X[:,1].reshape(-1,1)

clf.fit(X_new, y)
clf.predict_proba(list(X[1]))
# Its still giving the results, considering only the first element of the list
# array([[ 0.41601419,  0.23990973,  0.34407608]])

Dont think of X[1] as the list of features belonging to same sample. It can be any arbitrary list with a number of elements in it.

How is this handled here? Pardon me if this is a case of nit-picking.

[wxchan]

We only deal with np.ndarray, pd.DataFrame, scipy.sparse.csr_matrix and scipy.sparse.csc_matrix. In other case, we reshape data to scipy.sparse.csr_matrix first. You can see this part of codes: https://github.com/Microsoft/LightGBM/blob/6d4c7b03b7273f23b913406cc9c9d85d5178b373/python-package/lightgbm/basic.py#L408-L426.

We indeed didn't expect such behavior of scipy.sparse.csr_matrix, do you have better suggestions?

[vivekk0903]

Indeed, the output of

scipy.sparse.csr_matrix(data)

for the above mentioned data is an sparse matrix of shape (1,1), which when printed only has the first element of that list. So this looks like a problem.
Atleast, maybe a warning about the data conversion will certainly be more useful in my opinion. Something which states the shape of original and the converted data.

[guolinke]

@wxchan can we add some conversion in predict, before the overall conversion?

[wxchan]

@vivekk-ezdi Is the issue: You train with a n * 1 matrix, and predict with a 1 * 4 matrix, the prediction only uses first element of 1 * 4 matrix? Look like if you use 1 * 4 numpy array, the problem remains: clf.predict_proba(np.array([list(X[1])])). The issue is not checking predict input shape?

[vivekk0903]

@wxchan Ah, yes I did not think in that way. You are absolutely correct. Passing a 1-d array (either list or numpy) is not invoking any error or warning.
As its often said in scikit-learn, the predict() is performing too much magic trying to infer the data shape.

The earlier versions of scikit-learn used to give a warning when 1-d data is passed into it. And from version 19 onwards, they are giving a value error for it.

So something along the lines will be pretty good.

[wxchan]

@guolinke can we check the input shape for prediction in c++ part?

in python, we can:

1. use np.array for list, because it's dense representation
2. giving a warning for other data type

[guolinke]

@wxchan
It is hard to check in cpp side, since the parameters are pointers and it is hard to guess which is right format.

We can add the process for 1D list before this line: https://github.com/Microsoft/LightGBM/blob/master/python-package/lightgbm/basic.py#L384

[wxchan]

But the issue not only happens for 1D list or array, it happens for all data types when num of features are not same in training and prediction time.

[guolinke]

@wxchan I think it is not possible to let the number of features is exact the same. e.g. if input is one row CSR, we cannot infer its number of features, since the zero value are not stored.

[wxchan]

What if some prediction feature index bigger than max feature index of model?

[guolinke]

they will be ignored. you can find it from the code of predictor.