[python-package] Custom multiclass loss function doesn't work

[rosmineb]

Description

Hello,

I'm getting strange results for custom multiclass loss functions. I implemented multiclass logloss as a custom loss function, and trained while evaluating on 3 validation sets: the training data, the training data in shuffled order, and a heldout set. I see the training loss decrease for the train set, but it does not decrease for the training data in a shuffled order. The loss shouldn't depend on the order of samples in the dataset, so I would expect the training data and the shuffled training data should have the same loss.

As a sanity check, I used the same loss function for XGBoost, which was able to successfully train an accurate model.

I saw this issue: #1644
which suggested that the problem is 0 hessians. I tried the suggestion there and I still had the same problem.

Am I doing something wrong? Or is this a bug?
Thanks for your help!

Reproducible example

from sklearn.datasets import make_blobs
import lightgbm as lgb
from sklearn.model_selection import train_test_split
import numpy as np
import pdb
from scipy.special import softmax
from sklearn.metrics import accuracy_score
import random
import xgboost as xgb

C = 2
num_class = 4
# Data is Guassian blob in each of the 4 quadrants, numbered clockwise
# upper right: 0
# lower right: 1
# lower left: 2
# upper left: 3
X, Y = make_blobs(1000, n_features=2, centers=[(C, C), (C, -C), (-C, -C), (-C, C)])
X_train, X_val, y_train, y_val = train_test_split(X, Y, test_size=.2, random_state=0)
# for debugging, get the training data shuffled in a random order
reorder = list(range(X_train.shape[0]))
random.shuffle(reorder)
X_train_shuf = X_train[reorder, :]
y_train_shuf = y_train[reorder]

# code for loss function
class Loss(object):
    # Use https://maxhalford.github.io/blog/lightgbm-focal-loss/ as a template for a binary loss function
    def __init__(self, num_class, flatten_type='F', model_type='lgb'):
        self.num_class = num_class if num_class != 1 else 2
        self.flatten_type = flatten_type
        self.model_type = model_type

    def loss_function(self, y_true, raw_pred):
        # calculate probs as softmax of raw_pred, then get the correct index according to y_true
        p = self.softmax(raw_pred.reshape(y_true.shape[0], -1))
        p = np.clip(p, 1e-15, 1 - 1e-15)
        pt = p[np.arange(y_true.shape[0]), y_true.astype(int)]
        val = -1 * np.log(pt)
        return val

    def init_score(self, y_true):
        class_count = np.array([y_true[y_true == i].shape[0] for i in range(self.num_class)])
        output = np.ones((y_true.shape[0], self.num_class)) * class_count / np.sum(class_count)
        output = np.clip(output, 1e-15, 1 - 1e-15)
        return output.reshape(-1)

    def feval(self, preds, data):
        y_true = data.get_label()
        probs = preds.reshape((y_true.shape[0], -1))
        is_higher_better = False
        if self.model_type == 'lgb':
            return 'loss', self.loss_function(y_true, probs).mean(), is_higher_better
        else:
            return 'loss', self.loss_function(y_true, probs).mean()

    def predict(self, model, X, y_fit):
        return self.softmax(self.init_score(y_fit).reshape((-1, self.num_class))[0] + model.predict(X))

    def objective(self, raw_preds, data):
        y_true = data.get_label()
        probs = raw_preds.reshape((y_true.shape[0], -1))
        return self.grad(y_true, probs), self.hess(y_true, probs)

    def grad(self, y_true, y_pred):
        # To avoid errors in calculating derivatives, use numerical approximations
        func = lambda x : self.loss_function(y_true, x)
        grad = np.asarray([numerical_gradient_vectorized(func, point=y_pred, i=i) for i in range(self.num_class)])
        grad = grad.flatten(self.flatten_type)
        return grad

    def hess(self, y_true, y_pred):
        # To avoid errors in calculating derivatives, use numerical approximations
        func = lambda z: self.loss_function(y_true=y_true, raw_pred=z)
        diag_hess = [numerical_second_derivative_vectorized(func, y_pred, i, i) for i in range(self.num_class)]
        diag_hess = np.asarray(diag_hess)
        # adding 1e-6 is suggested to avoid 0 Hessians here: https://github.com/Microsoft/LightGBM/issues/1644
        # Although it does not seem to make an impact whether or not I add 1e-6
        return diag_hess.flatten(self.flatten_type) + 1e-6
    
    def softmax(self, z):
        return np.exp(z) / np.sum(np.exp(z), axis=1).reshape((-1, 1))
    
def numerical_second_derivative_vectorized(func, point, i, j, eps=.0001):
    # assume point is a row vector
    point = point.astype('float64') # without the higher precision, sometimes it fails
    ei = np.zeros_like(point).astype('float64')
    ej = np.zeros_like(point).astype('float64')
    ei[:, i] = eps
    ej[:, j] = eps
    return (func(point + ei + ej) - func(point+ ei) - func(point + ej) + func(point)) / eps ** 2

def numerical_gradient_vectorized(func, point, i, eps=1e-6):
    point = point.astype('float64')
    ei = np.zeros_like(point).astype('float64')
    ei[:, i] = eps
    return (func(point + ei) - func(point)) / eps

# Training LightGBM model
lgb_loss = Loss(num_class=num_class, model_type='lgb')
train_dataset = lgb.Dataset(X_train, y_train)
train_dataset_shuf = lgb.Dataset(X_train_shuf, y_train_shuf)
val_dataset = lgb.Dataset(X_val, y_val)
lgb_model = lgb.train(params={'n_estimators': 50, 'num_leaves': 300, 'num_class': num_class,
                          'min_data_in_leaf': 1},
                  train_set=train_dataset,
                  valid_sets=(train_dataset, train_dataset_shuf, val_dataset),
                  valid_names=('train', 'train_shuf', 'val'),
                  fobj=lgb_loss.objective,
                  feval=lgb_loss.feval)
# Training loss decreases for the 'train' set. However, it does not decrease for the 'train_shuf' set, which 
# is the same training data, just in shuffled order, which makes me think that this is a bug.

def lgb_predict(model, X, y_train, lgb_loss=lgb_loss):
    # Must add the initialization score before making prediction, see here:
    # https://maxhalford.github.io/blog/lightgbm-focal-loss/
    raw_preds = lgb_loss.predict(model, X, y_train)
    return np.argmax(raw_preds, axis=1)

lgb_preds_train = lgb_predict(lgb_model, X_train, y_train)
lgb_preds_train_shuf = lgb_predict(lgb_model, X_train_shuf, y_train)
lgb_preds_val = lgb_predict(lgb_model, X_val, y_train)

print('lgb accuracy train:', accuracy_score(y_train, lgb_preds_train))
print('lgb accuracy train_shuf:', accuracy_score(y_train, lgb_preds_train_shuf))
print('lgb accuracy val:', accuracy_score(y_val, lgb_preds_val))

# As a sanity check that I've implemented the loss function correctly, train with XGBoost
xgb_loss = Loss(num_class=num_class, model_type='xgb')

dtrain = xgb.DMatrix(X_train, y_train)
dtrain_shuf = xgb.DMatrix(X_train_shuf, y_train_shuf)
dval = xgb.DMatrix(X_val, y_val)
xmodel = xgb.train({'num_class': 4, 'disable_default_eval_metric': True},
                   dtrain,
                   num_boost_round=100,
                   obj=xgb_loss.objective) #,
                   #feval=xgb_loss.feval),
                   #evals=[(dtrain, 'train'), (dval, 'val')])

print('XGB train accuracy: ', accuracy_score(y_train, xmodel.predict(dtrain)))
print('XGB train_shuf accuracy: ', accuracy_score(y_train_shuf, xmodel.predict(dtrain_shuf)))
print('XGB val accuracy: ', accuracy_score(y_val, xmodel.predict(dval)))

Environment info

OS: Amazon Linux 2 (EC2 instance)
python: 3.7.10
LightGBM version or commit hash: 3.3.2

Command(s) you used to install LightGBM

pip install lightgbm -U

Additional Comments

Shortened output of code:
Notice how train loss decreases, but the loss on the shuffled training data and heldout set both increase.

[LightGBM] [Warning] Using self-defined objective function
[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.000355 seconds.
You can set force_col_wise=true to remove the overhead.
[LightGBM] [Info] Total Bins 510
[LightGBM] [Info] Number of data points in the train set: 800, number of used features: 2
[LightGBM] [Warning] Using self-defined objective function
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] train's loss: 1.01643 train_shuf's loss: 1.41085 val's loss: 1.44438
[2] train's loss: 0.779389 train_shuf's loss: 1.45818 val's loss: 1.49208
[3] train's loss: 0.610527 train_shuf's loss: 1.51741 val's loss: 1.539
[4] train's loss: 0.484318 train_shuf's loss: 1.58413 val's loss: 1.579
[5] train's loss: 0.387396 train_shuf's loss: 1.65603 val's loss: 1.62837
...
[48] train's loss: 0.000383055 train_shuf's loss: 4.8523 val's loss: 4.07682
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[49] train's loss: 0.000361902 train_shuf's loss: 4.88315 val's loss: 4.10556
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[50] train's loss: 0.00034246 train_shuf's loss: 4.91395 val's loss: 4.13448
lgb accuracy train: 0.23625
lgb accuracy train_shuf: 0.25
lgb accuracy val: 0.25

XGB train accuracy: 0.99
XGB train_shuf accuracy: 0.99
XGB val accuracy: 0.945

[jmoralez]

Hi @HotSauceTea, thank you for your interest in LightGBM. One possible error here is the reshaping:

For multi-class task, the preds is group by class_id first, then group by row_id. If you want to get i-th row preds in j-th class, the access way is score[j * num_data + i] and you should group grad and hess in this way as well (source)

So the predictions should be reshaped with order='F' to get the correct order and there may be some other detail in the loss function. Here's some sample code that shows it working correctly and might help you identify some other difference:

import random

import lightgbm as lgb
import numpy as np
from scipy.special import softmax
from sklearn.datasets import make_blobs
from sklearn.metrics import log_loss
from sklearn.model_selection import train_test_split

C = 2
num_class = 4
# Data is Guassian blob in each of the 4 quadrants, numbered clockwise
# upper right: 0
# lower right: 1
# lower left: 2
# upper left: 3
X, Y = make_blobs(1000, n_features=2, centers=[(C, C), (C, -C), (-C, -C), (-C, C)], random_state=0)
X_train, X_val, y_train, y_val = train_test_split(X, Y, test_size=.2, random_state=0)
# for debugging, get the training data shuffled in a random order
reorder = list(range(X_train.shape[0]))
random.shuffle(reorder)
X_train_shuf = X_train[reorder, :]
y_train_shuf = y_train[reorder]

def multiclass_custom_objective(y_pred, ds):
    y_true = ds.get_label()
    y_pred = y_pred.reshape((y_true.size, -1), order='F')
    num_rows, num_class = y_pred.shape
    prob = softmax(y_pred, axis=1)
    grad_update = np.zeros_like(prob)
    grad_update[np.arange(num_rows), y_true.astype(np.int32)] = -1.0
    grad = prob + grad_update
    factor = num_class / (num_class - 1)
    hess = factor * prob * (1 - prob)
    grad = grad.ravel(order='F')
    hess = hess.ravel(order='F')
    return grad, hess

def multiclass_metric(y_pred, ds):
    y_true = ds.get_label()
    y_pred = y_pred.reshape((y_true.size, -1), order='F')
    prob = softmax(y_pred, axis=1)
    return 'log-loss', log_loss(y_true, prob), False
    
params = {'objective': 'multiclass', 'n_estimators': 10, 'num_leaves': 7, 'num_class': num_class,'verbose': -1}
train_dataset = lgb.Dataset(X_train, y_train)
train_dataset_shuf = lgb.Dataset(X_train_shuf, y_train_shuf)
val_dataset = lgb.Dataset(X_val, y_val)

# Using LightGBM's built-in objective
lgb_model = lgb.train(params=params,
                  train_set=train_dataset,
                  valid_sets=(train_dataset, train_dataset_shuf, val_dataset),
                  valid_names=('train', 'train_shuf', 'val'),
                  callbacks=[lgb.log_evaluation(1)])
# [1]	train's multi_logloss: 1.12648	train_shuf's multi_logloss: 1.12648	val's multi_logloss: 1.12422
# [2]	train's multi_logloss: 0.938605	train_shuf's multi_logloss: 0.938605	val's multi_logloss: 0.934401
# [3]	train's multi_logloss: 0.794619	train_shuf's multi_logloss: 0.794619	val's multi_logloss: 0.789349
# [4]	train's multi_logloss: 0.680486	train_shuf's multi_logloss: 0.680486	val's multi_logloss: 0.674351
# [5]	train's multi_logloss: 0.588092	train_shuf's multi_logloss: 0.588092	val's multi_logloss: 0.582412
# [6]	train's multi_logloss: 0.512366	train_shuf's multi_logloss: 0.512366	val's multi_logloss: 0.507107
# [7]	train's multi_logloss: 0.449568	train_shuf's multi_logloss: 0.449568	val's multi_logloss: 0.443862
# [8]	train's multi_logloss: 0.397103	train_shuf's multi_logloss: 0.397103	val's multi_logloss: 0.390665
# [9]	train's multi_logloss: 0.352971	train_shuf's multi_logloss: 0.352971	val's multi_logloss: 0.346718
# [10]	train's multi_logloss: 0.315749	train_shuf's multi_logloss: 0.315749	val's multi_logloss: 0.309445

# Custom objective
lgb_model2 = lgb.train(params=params,
                  train_set=train_dataset,
                  valid_sets=(train_dataset, train_dataset_shuf, val_dataset),
                  valid_names=('train', 'train_shuf', 'val'),
                  fobj=multiclass_custom_objective,
                  feval=multiclass_metric,
                  callbacks=[lgb.log_evaluation(1)])
# [1]	train's log-loss: 1.12685	train_shuf's log-loss: 1.12685	val's log-loss: 1.12215
# [2]	train's log-loss: 0.938936	train_shuf's log-loss: 0.938936	val's log-loss: 0.932934
# [3]	train's log-loss: 0.794808	train_shuf's log-loss: 0.794808	val's log-loss: 0.78789
# [4]	train's log-loss: 0.680526	train_shuf's log-loss: 0.680526	val's log-loss: 0.673732
# [5]	train's log-loss: 0.588135	train_shuf's log-loss: 0.588135	val's log-loss: 0.581941
# [6]	train's log-loss: 0.512459	train_shuf's log-loss: 0.512459	val's log-loss: 0.506903
# [7]	train's log-loss: 0.449623	train_shuf's log-loss: 0.449623	val's log-loss: 0.44366
# [8]	train's log-loss: 0.397197	train_shuf's log-loss: 0.397197	val's log-loss: 0.390957
# [9]	train's log-loss: 0.353033	train_shuf's log-loss: 0.353033	val's log-loss: 0.34663
# [10]	train's log-loss: 0.315846	train_shuf's log-loss: 0.315846	val's log-loss: 0.309644

Let us know if this helps.

[Simon-Free]

Hello,

I'm dealing with the same problem, though I'm using the LGBM scikit learn wrapper.

I paid attention to the reshaping, but I don't know if my function does what is awaited by the package : why is everyone using a softmax ? Why is it necessary to clip things between 0 and 1 ?

Anyway, my model is not training (I have 3 classes, it only returns [.0, .0, .0] for all classes). I'm on lightgbm 3.3.1

def custom_objective(true, pred):
    y_true = np.array([np.array([(float(i == 0.0), float(i == 1.0), float(i == 2.0))]) for i in true])
    weight = np.array(
        [np.array([1., -.4, -.5]),
         np.array([-1., 1., -3.]),
         np.array([-1., -3., 1.]),]
    )
    grad = np.dot(y_true, weight)
    hess = np.zeros((len(true), 3))
    return grad.T.flatten(), hess.T.flatten()

model = LGBMClassifier(
    objective=custom_objective
)

During the training, I'm having at each iteration the following error:

  UserWarning,
[LightGBM] [Warning] Unknown parameter: disable_default_eval_metric
C:\Users\9605647W\AppData\Local\pypoetry\Cache\virtualenvs\kpiforecast--mVOw8nb-py3.7\lib\site-packages\lightgbm\sklearn.py:999: UserWarning: Cannot compute class probabilities or labels due to the usage of customized objective function.
Returning raw scores instead.
  _log_warning("Cannot compute class probabilities or labels "
C:\Users\9605647W\AppData\Local\pypoetry\Cache\virtualenvs\kpiforecast--mVOw8nb-py3.7\lib\site-packages\sklearn\model_selection\_validation.py:775: UserWarning: Scoring failed. The score on this train-test partition for these parameters will be set to nan. Details: 
Traceback (most recent call last):
  File "C:\Users\9605647W\AppData\Local\pypoetry\Cache\virtualenvs\kpiforecast--mVOw8nb-py3.7\lib\site-packages\sklearn\model_selection\_validation.py", line 762, in _score
    scores = scorer(estimator, X_test, y_test)
  File "C:\Users\9605647W\AppData\Local\pypoetry\Cache\virtualenvs\kpiforecast--mVOw8nb-py3.7\lib\site-packages\sklearn\metrics\_scorer.py", line 418, in _passthrough_scorer
    return estimator.score(*args, **kwargs)
  File "C:\Users\9605647W\AppData\Local\pypoetry\Cache\virtualenvs\kpiforecast--mVOw8nb-py3.7\lib\site-packages\sklearn\utils\metaestimators.py", line 113, in <lambda>
    out = lambda *args, **kwargs: self.fn(obj, *args, **kwargs)  # noqa
  File "C:\Users\9605647W\AppData\Local\pypoetry\Cache\virtualenvs\kpiforecast--mVOw8nb-py3.7\lib\site-packages\sklearn\pipeline.py", line 711, in score
    return self.steps[-1][1].score(Xt, y, **score_params)
  File "C:\Users\9605647W\AppData\Local\pypoetry\Cache\virtualenvs\kpiforecast--mVOw8nb-py3.7\lib\site-packages\sklearn\base.py", line 642, in score
    return accuracy_score(y, self.predict(X), sample_weight=sample_weight)
  File "C:\Users\9605647W\AppData\Local\pypoetry\Cache\virtualenvs\kpiforecast--mVOw8nb-py3.7\lib\site-packages\sklearn\metrics\_classification.py", line 206, in accuracy_score
    y_type, y_true, y_pred = _check_targets(y_true, y_pred)
  File "C:\Users\9605647W\AppData\Local\pypoetry\Cache\virtualenvs\kpiforecast--mVOw8nb-py3.7\lib\site-packages\sklearn\metrics\_classification.py", line 95, in _check_targets
    type_true, type_pred
ValueError: Classification metrics can't handle a mix of binary and multilabel-indicator targets

Thank you in advance 👍 !

[rosmineb]

Hi @jmoralez,
Thank you so much! That helped fix it!

[anitaschaefer]

I have the same issue as @Simon-Free: I use the Scikit-Learn API and y_pred only consists of zeros even though there are multiple classes. This is not the case if I use the LGBM API (then y_pred contains all classes). Is there any solution to this yet?

[github-actions]

This issue has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.