Description
According to https://github.com/IntelPython/scikit-learn_bench/blob/master/runner.py#L306-L310, the benchmark is said to be successful when 'daal4py' not in stderr, but when benchmark fails, stderr also contains 'daal4py' in some cases.
This issue may affect CI test, there are still errors although CIs are passed.
To reproduce, you can add print(stderr) under the if statement at https://github.com/IntelPython/scikit-learn_bench/blob/master/runner.py#L306, and run python runner.py --configs configs/testing/sklearn.json, then you will get errors like:
Traceback (most recent call last):
File "sklearn_bench/pca.py", line 66, in <module>
bench.run_with_context(params, main)
File "/home/zhaojieh/sklearn-benchmark-lulin/bench.py", line 567, in run_with_context
function()
File "sklearn_bench/pca.py", line 37, in main
fit_time, _ = bench.measure_function_time(pca.fit, X_train, params=params)
File "/home/zhaojieh/sklearn-benchmark-lulin/bench.py", line 267, in measure_function_time
return time_box_filter(func, *args,
File "/home/zhaojieh/sklearn-benchmark-lulin/bench.py", line 276, in time_box_filter
val = func(*args, **kwargs)
File "/home/zhaojieh/miniconda3/envs/sklearn-bench/lib/python3.8/site-packages/sklearn/decomposition/_pca.py", line 435, in fit
self._fit(X)
File "/home/zhaojieh/miniconda3/envs/sklearn-bench/lib/python3.8/site-packages/daal4py/sklearn/decomposition/_pca.py", line 260, in _fit
result = self._fit_full(X, n_components)
File "/home/zhaojieh/miniconda3/envs/sklearn-bench/lib/python3.8/site-packages/daal4py/sklearn/decomposition/_pca.py", line 153, in _fit_full
self._fit_full_daal4py(X, min(X.shape))
File "/home/zhaojieh/miniconda3/envs/sklearn-bench/lib/python3.8/site-packages/daal4py/sklearn/decomposition/_pca.py", line 142, in _fit_full_daal4py
self.n_samples_, self.n_features_ = n_samples, n_features
AttributeError: can't set attributeCASE sklearn,pca --data-format pandas --data-order F --dtype float64 --device none JSON DECODING ERROR:
Expecting value: line 1 column 1 (char 0)
I found this issue when developing #134, and you can check CI logs in this PR for more error messages.
This PR uses retcode to check whether there are errors when running benchmark and the error message can be displayed correctly, but the error in benchmark may need to be fixed separately.