about mode='Optuna'

[xuzhang5788]

even I installed it with @dev, I still can not use this option. Any suggestions?

[pplonski]

@xuzhang5788 I've updated the PyPi and master branches. You can use them.

To run in the Optuna mode:

automl = AutoML(
    algorithms=["Xgboost", "LightGBM", "CatBoost", "Random Forest", "Extra Trees", "Neural Networks"],
    mode="Optuna",
    optuna_time_budget=1200,
    total_time_limit=24*3600
)
automl.fit(X, y)

The optuna_time_budget=1200 means that 20 minutes will be used by Optuna to tune each algorithm. If there are 6 types of algorithms, then Optuna will use 6*20=120 minutes for all tuning. Then, the total_time_limit=24*3600 will be used for ML training (after tuning). In the Optuna mode, the tuning time and training time are separate.

The Optuna is not using feature engineering. To enable feature engineering:

automl = AutoML(
    algorithms=["Xgboost", "LightGBM", "CatBoost", "Random Forest", "Extra Trees", "Neural Networks"],
    mode="Optuna",
    optuna_time_budget=1200,
    total_time_limit=24*3600,
    kmeans_features=True,
    golden_features=True,
    mix_encoding=True,
    features_selection=True
)
automl.fit(X, y)

The example of Optuna mode on Kaggle TPS Apr-21 data: https://www.kaggle.com/mt77pp/mljar-automl-tps-apr-2021-manual-features

@xuzhang5788 please let me know if it works for you.

[xuzhang5788]

Sorry, it doesn't work with an error, TypeError: init() got an unexpected keyword argument 'optuna_time_budget'

[pplonski]

Please try to install the package in a fresh environment. It should work. When I install it in Kaggle kernel it is working.

[xuzhang5788]

Sorry. I reinstalled the package. When running cell 'from supervised.automl import AutoML', I got

RuntimeError Traceback (most recent call last)
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd

I reran that cell, errors disappeared, I ran

automl = AutoML(
algorithms=["Xgboost", "LightGBM", "CatBoost"],
mode="Optuna",
eval_metric='mse',
optuna_time_budget=1200,
total_time_limit=24*3600,
kmeans_features=True,
golden_features=True,
mix_encoding=True,
features_selection=True
)
automl.fit(X_train, y_train)

I got

AutoML directory: AutoML_13
Expected computing time:
Total training time: Optuna + ML training = 90000 seconds
Total Optuna time: len(algorithms) * optuna_time_budget = 3600 seconds
Total ML model training time: 86400 seconds
The task is regression with evaluation metric mse
AutoML will use algorithms: ['Xgboost', 'LightGBM', 'CatBoost']
AutoML will stack models
AutoML will ensemble availabe models

2021-04-03 23:31:54,586 supervised.exceptions ERROR No models produced.
Please check your data or submit a Github issue at https://github.com/mljar/mljar-supervised/issues/new.

AutoML steps: ['simple_algorithms', 'default_algorithms', 'insert_random_feature', 'features_selection', 'boost_on_errors', 'ensemble', 'stack', 'ensemble_stacked']
Skip simple_algorithms because no parameters were generated.
Skip default_algorithms because no parameters were generated.
Skip insert_random_feature because no parameters were generated.
Skip features_selection because no parameters were generated.
Skip boost_on_errors because no parameters were generated.

AutoMLException: No models produced.
Please check your data or submit a Github issue at https://github.com/mljar/mljar-supervised/issues/new.

Before my data was okay for other modes.

[pplonski]

@xuzhang5788 you are using Optuna mode correctly. There must be a bug or some other problem.

Could you please install the latest release code:

pip install -U mljar-supervised

and check how it runs with Compete and Optuna modes?

[xuzhang5788]

Sorry. It still doesn't work.
conda env with python 3.9.0, ubuntu 20.04

[pplonski]

@xuzhang5788 from your previous messages, it looks that MLJAR AutoML is running but can't fit your data. It produces some errors. Could you please post the code example with data?

[xuzhang5788]

Now, I have the following error when setting automal:
init() got an unexpected keyword argument 'optuna_time_budget'

[xuzhang5788]

My data is like
X_train

	0	1	2	3	4	5	6	7	8	9	...	1014	1015	1016	1017	1018	1019	1020	1021	1022	1023
K	V	L	G	S	G	A	F	G	T	...	0	0	1	1	0	1	0	1	1	0
K	P	L	G	E	G	A	F	G	Q	...	0	0	0	1	1	0	0	1	0	0
E	F	L	G	R	G	A	F	G	Q	...	0	0	0	0	0	0	0	1	0	1
R	V	L	G	K	G	G	Y	G	K	...	0	0	0	1	1	0	0	0	0	0
N	A	I	I	D	D	Y	K	V	K	...	1	1	0	0	0	0	1	0	0	0
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
R	V	L	G	K	G	G	Y	G	K	...	1	0	0	0	1	0	1	1	0	1
R	K	L	G	W	G	H	F	S	T	...	0	1	1	1	0	0	0	0	0	0
K	V	L	G	S	G	A	F	G	K	...	0	0	0	0	0	0	0	0	0	0
K	V	L	G	S	G	A	F	G	T	...	1	1	1	0	0	0	0	0	1	0
K	T	L	G	E	G	E	F	G	K	...	1	1	1	0	1	0	0	1	1	0

77312 rows × 1109 columns
from column[:85] are categorical variables, column[85:-1] are numerical variables only use 0 or 1. y_train is a numerical variable.

from supervised.automl import AutoML


automl = AutoML(ml_task='regression',
    algorithms=["Xgboost", "LightGBM", "CatBoost"],
    mode="Optuna",
    eval_metric='mse',
    optuna_time_budget=1200,
    total_time_limit=24*3600,
#    kmeans_features=True,
#    golden_features=True,
#    mix_encoding=True,
#    features_selection=True
               )
automl.fit(X_train, y_train)

I got
AutoML directory: AutoML_21
Expected computing time:
Total training time: Optuna + ML training = 90000 seconds
Total Optuna time: len(algorithms) * optuna_time_budget = 3600 seconds
Total ML model training time: 86400 seconds
The task is regression with evaluation metric mse
AutoML will use algorithms: ['Xgboost', 'LightGBM', 'CatBoost']
AutoML will stack models
AutoML will ensemble availabe models

2021-04-06 16:11:39,285 supervised.exceptions ERROR No models produced.
Please check your data or submit a Github issue at https://github.com/mljar/mljar-supervised/issues/new.

AutoML steps: ['simple_algorithms', 'default_algorithms', 'boost_on_errors', 'ensemble', 'stack', 'ensemble_stacked']
Skip simple_algorithms because no parameters were generated.
Skip default_algorithms because no parameters were generated.
Skip boost_on_errors because no parameters were generated.

AutoMLException: No models produced.
Please check your data or submit a Github issue at https://github.com/mljar/mljar-supervised/issues/new.

If I use mode='Compete', it worked well

[pplonski]

@xuzhang5788 is your data publicly available? So I can run locally? Could you please send me link to data? Or send data or some sample?

Do you get good models with Compete?

[xuzhang5788]

@pplonski Thank you.
My data is not publicly available, but I can give you. Please give me your email address or tell me how to send it to you.

[xuzhang5788]

@pplonski
Yes, I think I got good models with Compete mode.

[pplonski]

@xuzhang5788 please send data and code to piotr@mljar.com