Add prediction early stopping #550

cbecker · 2017-05-26T10:00:34Z

Adds classification prediction early stopping and necessary API. It is based on comparisons on the classification margin while prediction is performed.

cbecker · 2017-05-26T10:01:36Z

include/LightGBM/boosting.h

+  * \param earlyStop Early stopping instance
+  */
+  virtual void PredictRawEarlyStop(const double* features, double* output,
+                            const PredictionEarlyStopInstance& earlyStop) const = 0;


an early stopping struct instance is needed to predict a sample, and separates prediction early stopping from the configuration of GBDT, which I think helps keep the code an functionality separate.

cbecker · 2017-05-26T10:02:19Z

include/LightGBM/prediction_early_stop.h

+
+  /// Create an early stopping algorithm of type `type`, with given roundPeriod and margin threshold
+  LIGHTGBM_EXPORT PredictionEarlyStopInstance createPredictionEarlyStopInstance(const std::string& type,
+                                                                const PredictionEarlyStopConfig& config);


Possible types are none, multiclass and binary.

cbecker · 2017-05-26T10:03:27Z

src/boosting/prediction_early_stop.cpp

+        std::vector<double> votes(static_cast<size_t>(sz));
+        for (int i=0; i < sz; ++i)
+           votes[i] = pred[i];
+        std::partial_sort(votes.begin(), votes.begin() + 2, votes.end(), std::greater<double>());


Right now we're not verifying that the prediction vector is at least of size 2, @guolinke would it be ok to check for it here and throw an exception if this is not met?

This means that PredictRawEarlyStop would throw.

you can just use the margin of binary class when votes.size() == 1

guolinke · 2017-05-26T10:27:31Z

@cbecker
This is not a accuracy prediction, right? I mean it may drop the prediction accuracy.

cbecker · 2017-05-26T10:29:51Z

This is not a accuracy prediction, right? I mean it may drop the prediction accuracy.

Yes, indeed. It's something that is useful when speed is a concern. Compared to limiting the numbe rof trees used for prediction, using this type of early stopping 'adapts' automatically to the training instance.

guolinke · 2017-05-26T10:38:30Z

OK. BTW, do we really need to expose the callback? Or just letting user to set the roundPeriod and marginThreshold is enough ?

cbecker · 2017-05-26T12:19:02Z

OK. BTW, do we really need to expose the callback? Or just letting user to set the roundPeriod and marginThreshold is enough ?

I thought that exposing the callback is useful if the user wants to have a custom asymmetric early stopping method (e.g. if class 0 is background and 1 foreground, many place a threshold on the negative class score, and leave the positive one without early stopping. E.g. stop if binary score < -1.0)

cbecker · 2017-05-26T13:05:42Z

Also, do you know why the tests may be failing? There is an undefined reference to PredictRawEarlyStop but it looks strange to me, and it compiles successfully on my desktop (linux, gcc)

�[0m[ 51%] �[32mBuilding CXX object CMakeFiles/lightgbm.dir/src/boosting/gbdt_prediction.cpp.o
�[0m�[31m�[1mLinking CXX executable ../lightgbm
�[0mCMakeFiles/lightgbm.dir/src/boosting/gbdt.cpp.o:(.data.rel.ro._ZTVN8LightGBM4GBDTE[_ZTVN8LightGBM4GBDTE]+0xa0): undefined reference to `LightGBM::GBDT::PredictRawEarlyStop(double const*, double*, LightGBM::PredictionEarlyStopInstance const&) const'
CMakeFiles/lightgbm.dir/src/boosting/boosting.cpp.o:(.data.rel.ro._ZTVN8LightGBM4DARTE[_ZTVN8LightGBM4DARTE]+0xa0): undefined reference to `LightGBM::GBDT::PredictRawEarlyStop(double const*, double*, LightGBM::PredictionEarlyStopInstance const&) const'
CMakeFiles/lightgbm.dir/src/boosting/boosting.cpp.o:(.data.rel.ro._ZTVN8LightGBM4GOSSE[_ZTVN8LightGBM4GOSSE]+0xa0): undefined reference to `LightGBM::GBDT::PredictRawEarlyStop(double const*, double*, LightGBM::PredictionEarlyStopInstance const&) const'
collect2: error: ld returned 1 exit status
make[2]: *** [../lightgbm] Error 1

guolinke · 2017-05-26T13:16:34Z

@cbecker refer to this pr #482 . It will replace the gbdt_prediction.cpp file. I think you may need to write your prediction code in a new cpp file.

cbecker · 2017-05-26T13:32:27Z

@cbecker refer to this pr #482 . It will replace the gbdt_prediction.cpp file. I think you may need to write your prediction code in a new cpp file.

Thanks, that looks a bit tricky to me, the fact that it's overwriting its own code. I'll put it in a separate file, at least for now it will do, but it won't be able to deal with pre-compiled models. Btw, which speed up do you achieve by having the model in if/else c++ statements?

wxchan · 2017-05-26T13:37:05Z

I think you can put your code in gbdt.cpp.

cbecker · 2017-05-26T13:57:03Z

I think you can put your code in gbdt.cpp.

For me there or in the current file works too. Let me know what you prefer and I'll make the changes and rename the commit, if we can merge 👍

cbecker · 2017-05-26T15:13:33Z

Done now :)

guolinke · 2017-05-27T02:14:30Z

@cbecker It seems this features cannot be easy to use. User must write his own cpp code to call this function ?

cbecker · 2017-05-28T06:58:05Z

@cbecker It seems this features cannot be easy to use. User must write his own cpp code to call this function ?

True, I will add the respective C api tomorrow and get back to you.

cbecker · 2017-05-29T08:13:01Z

@guolinke would you agree with removing the parallel loops in PredictRaw() and Predict() now? Because then the code can be modularized and kept much simpler, and we'll have most of the prediction codein single function instead of 3 of them (including early stopping).

I can do this in a commit in this PR

guolinke · 2017-05-29T08:19:11Z

@cbecker
OK, you can make them to single thread.

guolinke · 2017-05-29T08:20:17Z

@cbecker
BTW, can you also support this for the CLI version?

cbecker · 2017-05-29T09:40:23Z

Tests won't pass yet, I have to modify ModelToIfElse() first, but the overall functionality is there now.

cbecker · 2017-05-29T10:27:13Z

@guolinke I am wondering why the new two C apis are not being exported to the .so file, have you seen this before? Are you stripping symbols at some stage, and if so, how do you control which ones to keep?

EDIT: I think it may be that my definition is not in extern C. I will check.

guolinke · 2017-05-29T11:17:50Z

src/c_api.cpp

@@ -162,6 +163,7 @@ class Booster {

  void Predict(int num_iteration, int predict_type, int nrow,
               std::function<std::vector<std::pair<int, double>>(int row_idx)> get_row_fun,
+               PredictionEarlyStoppingHandle early_stop_handle,


I think we cannot use function_pointer in c_api. It is hard to use this in python/R

Oh, I am wrong. it seems you use handle. So it is a class.

guolinke · 2017-05-29T11:31:50Z

@cbecker
https://github.com/cbecker/LightGBM/blob/4e3fadb31c0e1213f57b8661456fbc69bf8df5de/src/c_api.cpp#L239-L261

I think you should these two apis outside of this class.

wxchan · 2017-05-29T13:05:42Z

python-package/lightgbm/basic.py

+        Parameters
+        ----------
+        early_stop_type: string
+            "none", "binary" or "multiclass". Regression is not supported.


I think you can just use None instead of string 'none' here

Never mind, seem "none" is better

Thanks, I allow for both options now.

cbecker · 2017-05-29T13:15:28Z

@guolinke Can we also support this in CLI version ? By passing the parameters when running LightGBM. And we can create an singleton Handle in CLI version, So that we don't need to create the instance of handle every time calling the prediction (https://github.com/cbecker/LightGBM/blob/5f50904dceec524451e17bae1ac7308ab180fc50/src/boosting/gbdt_prediction.cpp#L22-L26) .

I am not sure where to modify the code for that. I think it'd be safer if you do that, as I am not familiar with the CLI code. Let me know if you agree with the state of the current PR.

On my side I am getting a 2x speed up with very little loss in accuracy in one of my tests. I am classifying millions of samples, and I got down from 1 hour to 1/2 an hour :)

wxchan · 2017-05-29T13:20:24Z

I think it's ready to merge for python and if-else part. You can add an example in examples/python-guide/advanced_example.py if you want.

wxchan · 2017-05-29T13:25:48Z

by the way, you need to update windows folder for new files. it's easy to forget.

guolinke · 2017-05-29T13:26:54Z

@cbecker OK, i can do it.
BTW, I update your code directly for the parallel by samples.

cbecker · 2017-05-29T13:32:09Z

by the way, you need to update windows folder for new files. it's easy to forget.

I'm not sure what this means, is it something I have to do?

guolinke · 2017-05-29T13:35:57Z

@cbecker
I think wxchan's point is update the lightgbm.vcproj in the windows folder.

wxchan · 2017-05-29T13:38:54Z

yes, LightGBM.vcxproj and LightGBM.vcxproj.filters

…rly stopping

cbecker · 2017-05-29T13:47:40Z

yes, LightGBM.vcxproj and LightGBM.vcxproj.filters

Thanks, I edited the file and made the changes manually as I don't have MSVC here, let me know if there are any issues.

…therwise

guolinke · 2017-05-29T14:23:11Z

@wxchan Do you have any other comments ?

wxchan · 2017-05-29T14:36:46Z

@guolinke no.

ddDragon · 2017-06-05T15:13:26Z

I used the early stopping parameter about a month ago, I found that didn't work? so this version is fixed the issue? or May be the method that i used was wrong?

guolinke · 2017-06-05T15:15:27Z

@ddDragon This is the early stopping for the prediction, not the training.

And I think the early stopping for training is always working. Maybe you use it by the wrong method.

ddDragon · 2017-06-06T02:38:59Z

I used the parameters in lgb.train api named early_stopping_rounds. I set 30 rounds early stopping, total 5000 iterations training. I found the best valid set score was at about 2000 rounds. But all times the model trained all the 5000 rounds? This confused me a lot.

guolinke · 2017-06-06T02:40:46Z

@wxchan any idea about that ?
@ddDragon can you provide the re-produce script ?

ddDragon · 2017-06-06T05:33:34Z

So sorry that I did't backup my code. Maybe something wrong in my code. If I meet this problem next time I would save it. Lgb is mush faster than xgb without loss accuracy. Very nice tool!

* Add early stopping for prediction * Fix GBDT if-else prediction with early stopping * Small C++ embelishments to early stopping API and functions * Fix early stopping efficiency issue by creating a singleton for no early stopping * Python improvements to early stopping API * Add assertion check for binary and multiclass prediction score length * Update vcxproj and vcxproj.filters with new early stopping files * Remove inline from PredictRaw(), the linker was not able to find it otherwise

msftclas added the cla-already-signed label May 26, 2017

cbecker commented May 26, 2017

View reviewed changes

cbecker force-pushed the earlyStopping branch from fdbdbb4 to 6a14ccf Compare May 26, 2017 13:35

cbecker force-pushed the earlyStopping branch from 6a14ccf to 145bb44 Compare May 26, 2017 15:12

guolinke closed this May 29, 2017

guolinke reopened this May 29, 2017

msftclas added the cla-already-signed label May 29, 2017

cbecker force-pushed the earlyStopping branch 2 times, most recently from bd4ff80 to 1a8e807 Compare May 29, 2017 09:37

guolinke reviewed May 29, 2017

View reviewed changes

wxchan reviewed May 29, 2017

View reviewed changes

Carlos Becker added 7 commits May 29, 2017 15:39

Add early stopping for prediction

3fad89b

Fix GBDT if-else prediction with early stopping

779074f

Small C++ embelishments to early stopping API and functions

4b92052

Fix early stopping efficiency issue by creating a singleton for no ea…

dbf06f0

…rly stopping

Python improvements to early stopping API

5f51604

Add assertion check for binary and multiclass prediction score length

cc504b3

Update vcxproj and vcxproj.filters with new early stopping files

e0ed7bf

cbecker force-pushed the earlyStopping branch from 6c21e73 to e0ed7bf Compare May 29, 2017 13:47

Remove inline from PredictRaw(), the linker was not able to find it o…

37c8890

…therwise

guolinke merged commit 993bbd5 into microsoft:master May 29, 2017

guolinke mentioned this pull request Apr 1, 2018

Just questions #1293

Closed

guolinke mentioned this pull request Aug 22, 2018

PredictionEarlyStopInstance questions #1574

Closed

lock bot locked as resolved and limited conversation to collaborators Mar 12, 2020

Add prediction early stopping #550

Add prediction early stopping #550

Conversation

cbecker commented May 26, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

guolinke commented May 26, 2017

cbecker commented May 26, 2017

guolinke commented May 26, 2017 • edited Loading

cbecker commented May 26, 2017

cbecker commented May 26, 2017

guolinke commented May 26, 2017

cbecker commented May 26, 2017

wxchan commented May 26, 2017

cbecker commented May 26, 2017

cbecker commented May 26, 2017

guolinke commented May 27, 2017

cbecker commented May 28, 2017

cbecker commented May 29, 2017 • edited Loading

guolinke commented May 29, 2017

guolinke commented May 29, 2017 • edited Loading

cbecker commented May 29, 2017

cbecker commented May 29, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

guolinke commented May 29, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cbecker commented May 29, 2017

wxchan commented May 29, 2017

wxchan commented May 29, 2017

guolinke commented May 29, 2017

cbecker commented May 29, 2017

guolinke commented May 29, 2017

wxchan commented May 29, 2017

cbecker commented May 29, 2017

guolinke commented May 29, 2017

wxchan commented May 29, 2017

ddDragon commented Jun 5, 2017

guolinke commented Jun 5, 2017

ddDragon commented Jun 6, 2017

guolinke commented Jun 6, 2017

ddDragon commented Jun 6, 2017

cbecker commented May 26, 2017 •

edited

Loading

guolinke commented May 26, 2017 •

edited

Loading

cbecker commented May 29, 2017 •

edited

Loading

guolinke commented May 29, 2017 •

edited

Loading

cbecker commented May 29, 2017 •

edited

Loading

guolinke commented May 29, 2017 •

edited

Loading