[BUG] TuneBaseSearchCV: Multiple scores as dictionary

[felipemonroy]

Hi everyone,

I am having issues when using TuneSearch CV and a dictionary of scores. The dictionary is the following:

from sklearn.metrics import make_scorer, cohen_kappa_score

{
    'kappa': make_scorer(cohen_kappa_score), 
    'kappa_quadratic': make_scorer(cohen_kappa_score, weights="quadratic")
}

The problem is in the method _format_results of the TuneBaseSearchCV class. Specifically, when creating the test_scores and train_scores dictionaries. Here is the code:

        test_scores = {}
        train_scores = {}
        for name in self.scoring_:
            test_scores[name] = [
                df[
                    [
                        col
                        for col in dfs[0].columns
                        if "split" in col and "test_%s" % name in col
                    ]
                ].to_numpy()
                for df in finished
            ]

So, when doing if "split" in col and "test_%s" % name in col for the score kappa the results of both scores (kappa and kappa_quadratic) are returned, causing an error downstream.

Now I am using scores names that avoid that confusion, but it took me a while to figure it out.

ray: 1.13.0
tune-sklearn: 0.4.3