torchmetrics

[yutaro-s]

Hi, thank you for sharing your code.

I am wondering if there is a bug in the evaluation of the Propaganda Meme dataset.
In the evaluation procedure, torchmetrics.F1 receives preds (and batch['labels']), but the data type of preds is Int (long).

hateclipper/engine.py

Line 395 in 04c86b8

output['f1'] = self.f1(preds, batch['labels'])

Given an Int tensor, TorchMetrics returns unexpected results as below:

>>> import torch
>>> from torchmetrics import F1Score
>>> target = torch.tensor([0, 1, 1, 0])
>>> preds = torch.tensor([0, 1, 1, 1])
>>> f1 = F1Score(mdmc_average="global")
>>> f1(preds, target)
tensor(0.7500)
>>> f1(preds.long(), target)
tensor(0.7500)
>>> f1(preds.float(), target)
tensor(0.8000)

I tested this with PyTorch v1.13.1 and TorchMetrics v0.10.0.

I didn't check the other metrics (e.g., precision and recall), but the same problem may exist.

Thank you.