val_sample_weight not used in calculation of custom validation metrics

[kevinkotzen]

Moving this here from tensorflow/tensorflow#51902
A colab from @sanatmpa1 replicating issue in tf v2.5,v2.6 and nightly is gist.

System information

• Custome code using standard keras and tfa.metrics
• Operating System: CentOS Linux 7 (Core), Kernel: Linux 3.10.0-1160.el7.x86_64
• Tensorflow installed with pip install tf-nightly-gpu
• v1.12.1-63317-g034300c177d 2.7.0-dev20210907
• NVIDIA-SMI 465.19.01 Driver Version: 465.19.01 CUDA Version: 11.3
• A100-40GB

Describe the current behavior
All validation metrics other than loss (set in model.compile(metrics=['accuracy', 'another_metrics'])) and reported during model.fit(......,val_data=(x_val, y_val, val_sample_weight)) do not seem to take the val_sample_weight into consideration. The loss metrics works as expected.

Describe the expected behavior
The val_sample_weight associated with the x_val and y_val should be used in the calculation of 'accuracy' and other metrics such as tfa.metrics.CohenKappa. This is important for early stopping of model based on validation metrics.

Contributing

• Do you want to contribute a PR? (yes/no): no
• Briefly describe your candidate solution(if contributing):

Standalone code to reproduce the issue
import os
import numpy as np
import tensorflow as tf
import tensorflow_addons as tfa

inputs = tf.keras.Input(shape=(10))
x = tf.keras.layers.Dense(100, activation='relu')(inputs)
x = tf.keras.layers.Dense(10, activation='relu')(x)
outputs = tf.keras.layers.Dense(3, activation='softmax')(x)
model = tf.keras.models.Model(inputs=inputs, outputs=outputs)
model.compile('adam', loss='sparse_categorical_crossentropy', metrics=['accuracy', tfa.metrics.CohenKappa(num_classes=3, sparse_labels=True)], sample_weight_mode='temporal')
model.summary()
X =tf.random.uniform(shape=[100,10], minval=0, maxval=1, dtype=tf.float32)
Y = tf.random.uniform(shape=[100,1], minval=0, maxval=3, dtype=tf.int64)
W = tf.ones([100])
W2 = tf.zeros([100])
model.fit(X, Y, sample_weight=W, validation_data=(X,Y,W2), epochs=5, verbose=1)
probs = model.predict(X)
kappa = tfa.metrics.CohenKappa(num_classes=3, sparse_labels=True)
kappa.update_state(Y, probs)
print("Kappa Without Weights", kappa.result())
kappa = tfa.metrics.CohenKappa(num_classes=3, sparse_labels=True)
kappa.update_state(Y, probs,W2)
print("Kappa With Weights", kappa.result())

Other info / logs Include any logs or source code that would be helpful to

W2 is all zeros. Notice that the val_loss is 0.000 as expected, but the val_accuracy and val_kappa are not. See prints the end showing kappa behavior with and without weights.

Epoch 1/5
4/4 [==============================] - 1s 215ms/step - loss: 1.0857 - accuracy: 0.4000 - cohen_kappa: -0.0130 - val_loss: 0.0000e+00 - val_accuracy: 0.3900 - val_cohen_kappa: -0.0281
Epoch 2/5
4/4 [==============================] - 0s 31ms/step - loss: 1.0766 - accuracy: 0.4000 - cohen_kappa: -0.0096 - val_loss: 0.0000e+00 - val_accuracy: 0.4000 - val_cohen_kappa: -0.0052
Epoch 3/5
4/4 [==============================] - 0s 30ms/step - loss: 1.0736 - accuracy: 0.3900 - cohen_kappa: -0.0214 - val_loss: 0.0000e+00 - val_accuracy: 0.4000 - val_cohen_kappa: -0.0059
Epoch 4/5
4/4 [==============================] - 0s 31ms/step - loss: 1.0706 - accuracy: 0.4000 - cohen_kappa: -0.0037 - val_loss: 0.0000e+00 - val_accuracy: 0.4000 - val_cohen_kappa: -0.0059
Epoch 5/5
4/4 [==============================] - 0s 29ms/step - loss: 1.0666 - accuracy: 0.4000 - cohen_kappa: -0.0081 - val_loss: 0.0000e+00 - val_accuracy: 0.4200 - val_cohen_kappa: 0.0213
Kappa Without Weights tf.Tensor(0.021262169, shape=(), dtype=float32)
Kappa With Weights tf.Tensor(0.0, shape=(), dtype=float32)

[haifeng-jin]

May need to verify with other custom metrics.
The sample_weights is handled by each metric, not by Keras framework.
Keras doesn't treat custom metrics differently than built-in metrics.
Likely to be a problem in the implementation of the custom metric.

[kevinkotzen]

Hi @haifeng-jin,
Thanks for your reply.
The built-in metrics 'accuracy' and 'sparse_categorical_accuracy' seem to also have the same problem. In fact, testing shows that NO weights are passed to all metrics (other than loss).

Try the following example metric in this [gist]. (https://colab.research.google.com/gist/tilakrayal/93668cb75a71299e9d335249a7450ca2/un51902.ipynb)

class TestSampleWeights(tf.keras.metrics.Metric):
  """
  Simply sums sample weights and returns that as the metric. 
  """
  def __init__(self, name='test_sample_weight', **kwargs):
    super(TestSampleWeights, self).__init__(name=name, **kwargs)
    self.sample_weight_sum = -1

  def update_state(self, y_true, y_pred, sample_weight=None):
    if sample_weight is None:
      self.sample_weight_sum=-2
    else:
      self.sample_weight_sum = tf.math.reduce_sum(sample_weight)

  def result(self):
    return self.sample_weight_sum

  def reset_states(self):
    self.sample_weight_sum = -1

Note: I had initially thought that sample weights were working for train data, but after more testing, this is not the case.

[3shv]

Don't you need to use weighted_metrics argument in compile args? See this #7536
I found this info hard to find as well. It's a bit strange that keras requires a special argument for this and completely ignores the sample weight for metrics without giving a proper warning.

[kevinkotzen]

@Rajesh-Veeranki thank you for your input. The solution you proposed is correct. setting weighted metrics=[val_metric_to_weight] works.

I agree that this is confusing and a warning should be raised if using sample/class weighting and validation data. I will update this from a bug report to a feature request. Thanks again.