MSE is negative when returned by cross_val_score

[tdomhan]

The Mean Square Error returned by sklearn.cross_validation.cross_val_score is always a negative. While being a designed decision so that the output of this function can be used for maximization given some hyperparameters, it's extremely confusing when using cross_val_score directly. At least I asked myself how a the mean of a square can possibly be negative and thought that cross_val_score was not working correctly or did not use the supplied metric. Only after digging in the sklearn source code I realized that the sign was flipped.

This behavior is mentioned in make_scorer in scorer.py, however it's not mentioned in cross_val_score and I think it should be, because otherwise it makes people think that cross_val_score is not working correctly.

[jaquesgrobler]

You're referring to

greater_is_better : boolean, default=True

Whether score_func is a score function (default), meaning high is good, 
or a loss function, meaning low is good. In the latter case, the scorer 
object will sign-flip the outcome of the score_func.

in http://scikit-learn.org/stable/modules/generated/sklearn.metrics.make_scorer.html
? (just for reference's sake)

I agree that that it can be more clear in cross_val_score docs

Thanks for reporting

[ogrisel]

Indeed we overlooked that issue when doing the Scorer refactoring. The following is very counter-intuitive:

>>> import numpy as np
>>> from sklearn.datasets import load_boston
>>> from sklearn.linear_model import RidgeCV
>>> from sklearn.cross_validation import cross_val_score

>>> boston = load_boston()
>>> np.mean(cross_val_score(RidgeCV(), boston.data, boston.target, scoring='mean_squared_error'))
-154.53681864311497

/cc @larsmans

[ogrisel]

BTW I don't agree that it's a documentation issue. It's cross_val_score should return the value with the sign that matches the scoring name. Ideally the GridSearchCV(*params).fit(X, y).best_score_ should be consistent too. Otherwise the API is very confusing.

[tdomhan]

I also agree that a change to return the actual MSE without the sign switched would be the way better option.

The scorer object could just store the greater_is_better flag and whenever the scorer is used the sign could be flipped in case it's needed, e.g. in GridSearchCV.

[larsmans]

I agree that we have a usability issue here, but I don't fully agree with @ogrisel's solution that we should

return the value with the sign that matches the scoring name

because that's an unreliable hack in the long run. What if someone defines a custom scorer with a name such as mse? What if they do follow the naming pattern but wrap the scorer in a decorator that changes the name?

The scorer object could just store the greater_is_better flag and whenever the scorer is used the sign could be flipped in case it's needed, e.g. in GridSearchCV.

This is what scorers originally did, during development between the 0.13 and 0.14 releases and it made their definition a lot harder. It also made the code hard to follow because the greater_is_better attribute seemed to disappear in the scorer code, only to reappear in the middle of the grid search code. A special Scorer class was needed to do something that ideally, a simple function would do.

I believe that if we want to optimize scores, then they should be maximized. For the sake of user-friendlyness, I think we might introduce a parameter score_is_loss ∈ ["auto", True, False] that only changes the display of scores and can use a heuristic based on the built-in names.

[larsmans]

That was a hurried response because I had to get off the train. What I meant by "display" is really the return value from cross_val_score. I think scorers should be simple and uniform and the algorithms should always maximize.

This does introduce an asymmetry between built-in and custom scorers.

Ping @GaelVaroquaux.

[jaquesgrobler]

I like the score_is_loss solution, or something to that effect.. the sign change to match the scoring name seems hard to maintain could cause problems as @larsmans mentioned

[tdomhan]

what's the conclusion, which solution should we go for? :)

[amelio-vazquez-reina]

@tdomhan @jaquesgrobler @larsmans Do you know if this applies to r2 as well? I am noticing that the r2 scores returned by GridSearchCV are also mostly negative for ElasticNet, Lasso and Ridge.

[larsmans]

R² can be either positive or negative, and negative simply means your model is performing very poorly.

[jnothman]

IIRC, @GaelVaroquaux was a proponent of returning a negative number when greater_is_better=False.

[larsmans]

r2 is a score function (greater is better), so that should be positive if your model is any good -- but it's one of the few performance metrics that can actually be negative, meaning worse than 0.

[mblondel]

What is the consensus on this issue? In my opinion, cross_val_score is an evaluation tool, not a model selection one. It should thus return the original values.

I can fix it in my PR #2759, since the changes I made make it really easy to fix. The trick is to not flip the sign upfront but, instead, to access the greater_is_better attribute on the scorer when doing grid search.