[WIP][java][mmlspark] speed up inference by ~20% in java layer

[imatiach-msft]

Speed up inference by returning java array from LGBM_BoosterPredictForMatSingle and LGBM_BoosterPredictForCSRSingle. This reduces JNI calls when getting the values from the array and also removes one step to allocate the output array in the Java side, removing another JNI call.

Testing in mmlspark, time was reduced from 82159416923ns to 65852779813ns, ~20% improvement.

FYI @eisber

[imatiach-msft]

looks like I need to delete before returning here in case of error

delete[] out_result;

[eisber]

can you use

vector<double> data0;
data0.resize(*out_len);

this will get you RAII behavior for errors (exceptions,...)

[imatiach-msft]

great idea

[AlbertoEAF]

One question @imatiach-msft - why not reuse the array instead of allocating a new one each time?

I have another extremely fast high-throughput low-latency implementation that is already in production and uses this functions and instead reuses the array *out_result for every call, meaning you only have to read the values, and no allocations are necessary. Isn't this new change going to make it slower than that solution?

[imatiach-msft]

@AlbertoEAF great point, I am investigating that as well, note this is still WIP. Sorry haven't had more time to look into this lately. I did discover an issue in how I was calculating the speedup, so this isn't quite ready yet.

[jameslamb]

Whenever you have time, can you please merge in master? Some changes have been made to our CI jobs to fix failures there. I apologize for the inconvenience.

[guolinke]

@imatiach-msft
What is the status of this PR?
do you have a plan for it?

[imatiach-msft]

@guolinke yes, I will work on it again when I have time, let me close it for now

[github-actions]

This pull request has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.