Add weighted_metrics arg to compile

[nicolewhite]

Alternative to #7482.

This implementation is growing on me. I think the flexibility here is nice.

from keras.models import Sequential
from keras.layers import Dense

import numpy as np

X = np.random.normal(size=(100, 10))
y = np.random.randint(2, size=100)
sample_weight = np.random.normal(size=100)
loss = 'binary_crossentropy'

model = Sequential()
model.add(Dense(10, input_shape=(10, )))
model.add(Dense(1, activation='sigmoid'))

model.compile(loss=loss, optimizer='rmsprop', metrics=[loss], weighted_metrics=[loss])

model.fit(X, y, epochs=5, verbose=2, sample_weight=sample_weight)

Epoch 1/5
0s - loss: -1.1340e-01 - binary_crossentropy: 0.7633 - weighted_binary_crossentropy: -1.1340e-01
Epoch 2/5
0s - loss: -1.2764e-01 - binary_crossentropy: 0.7729 - weighted_binary_crossentropy: -1.2764e-01
Epoch 3/5
0s - loss: -1.3798e-01 - binary_crossentropy: 0.7782 - weighted_binary_crossentropy: -1.3798e-01
Epoch 4/5
0s - loss: -1.4393e-01 - binary_crossentropy: 0.7837 - weighted_binary_crossentropy: -1.4393e-01
Epoch 5/5
0s - loss: -1.5062e-01 - binary_crossentropy: 0.7877 - weighted_binary_crossentropy: -1.5062e-01

[fchollet]

Could you list a few use cases where this approach is better? This would be useful data for picking one API or the other.

[nicolewhite]

I think this approach is better because the flexibility covers both the use cases:

• someone wants to track both an unweighted and weighted metric
• someone wants to track only a weighted metric

The other API option, simply setting weigh_metrics=True, only allows for the latter.

You might be interested in tracking both a weighted and unweighted metric when you are optimizing for some weighted sub-problem that you have devised, but you still want to make sure the model is performing its initial job well, that is, its accuracy regardless of the weights (assuming a classification problem). It also seems likely you'd want to understand the relationship between the performance of the weighted and unweighted metrics as you tinker with your sample weights. For example, it might be the case that you are choosing weights that perform really well on your weighted metric but your unweighted metric tanks, which could raise red flags and cause you to reconsider your approach.

It also seems possible in the multi-output scenario that someone would want to compute a weighted metric for one output and an unweighted metric for another. This API would handle that use case very easily, whereas the other would not. For example:

metrics = {'output_1': ['accuracy']}
weighted_metrics = {'output_2': ['accuracy']}

model.compile(..., metrics=metrics, weighted_metrics=weighted_metrics)

I should probably add a test for this use case, actually.

[fchollet]

I agree this PR is the better approach, so we will go with this one.

[fchollet]

Style: lines in the test are too long, please shorten.

[fchollet]

Fix indentation

[fchollet]

Why?

[nicolewhite]

CNTK is only accurate to two decimal places. I'm not sure why.

[fchollet]

LGTM, thanks